acceptodds
Under review as a conference paper at ICLR 2027

Beyond Ego Privilege: Queryable Multi-Agent 4D Occupancy World Model

Abstract

Ego-conditioned occupancy world models typically tie both motion conditioning and prediction windows to ego. In cooperative driving, comparing plans shared by multiple vehicles requires assessing their joint effects in the same road region, but not in a window that changes with each ego plan. We present BEP-World (Beyond Ego Privilege), a queryable 4D occupancy world model with a shared plan interface for ego and surrounding vehicles and separate spatial queries. Besides, we propose Query-Aligned Memory Aggregation and Grounded Plan Fusion to organize historical evidence and vehicle plans in the requested coordinates before scene dynamics, enabling direct forecasting without post-prediction warping. On OpenOccupancy, BEP-World supports forecasting, querying, multi-agent conditioning, and their combination. It predicts occupancy beyond the future ego window and benefits from surrounding-vehicle plans under non-ego queries. Furthermore, we construct BEP-Pairs, a simulation-based paired-intervention development set for diagnosing responses beyond controlled vehicles. Extensive experiments and ablations identify spatial fields as the main source of plan utilization and reveal trade-offs between forecasting accuracy, query coverage, and conditioning gains. Code are available at https://anonymous.4open.science/r/BEP-World.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Occupancy, World Model, Automatic Driving, Interactive, Multi-agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_e9kgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BEP-World, a 4D occupancy model that separates vehicle-plan conditioning from spatial querying. Query-Aligned Memory Aggregation (QAMA) aligns historical scene features with requested query coordinates, while Grounded Plan Fusion (GPF) incorporates plans for ego and surrounding vehicles. Experiments on OpenOccupancy cover ego forecasting, spatial querying, multi-agent conditioning, and their combination. The authors also introduce BEP-Pairs, a CARLA-based paired-intervention dataset. The results show that joint plan conditioning and spatial querying are feasible, although the evidence for predicting responses of unconditioned vehicles is limited.

Critical strengths
  1. The formulation makes plan comparisons more flexible.

The paper separates which vehicle plans are provided from where future occupancy is queried. This lets ego and surrounding vehicles share a common conditioning interface while evaluating different plans over the same spatial region. The formulation addresses a limitation of ego-centric setups and could be useful for cooperative driving and plan evaluation.

  1. The evaluation includes targeted interventions and informative ablations.

The S0–S3 protocol separates ego forecasting, spatial querying, multi-agent conditioning, and their combination. Matched-checkpoint comparisons and per-seed results help assess the contribution of additional vehicle plans. BEP-Pairs also distinguishes changes to controlled vehicles from responses of unconditioned vehicles. The GPF ablations favor spatial fields over agent tokens alone, supporting the use of explicit geometric correspondence.

Critical weaknesses
  1. The paper does not establish why joint modeling is preferable to a modular alternative.

Prior work has explored multi-agent conditioning and spatial querying. BEP-World combines them in one interface, but the paper does not compare it with a modular multi-agent predictor followed by spatial querying. Without a comparison using matched inputs and computational budgets, it is difficult to tell whether the joint formulation offers a distinct modeling benefit or mainly combines existing capabilities.

  1. The experiments do not fully isolate the claimed query and interaction capabilities.

The Query-versus-Warp comparison favors direct querying in novel regions, but Warp cannot predict outside its original ego-centered window. Under S3, direct querying also has lower full-window mIoU than Warp (37.75 versus 38.56), and there is no direct QAMA ablation or spatial-coverage-matched baseline. On BEP-Pairs, the use of factual future trajectories makes it difficult to separate interaction modeling from trajectory following: controlled IoU improves from 48.90 to 60.39, but responder-change F1 rises only from 8.65 to 11.12, with 10.03% recall. Some query windows are also defined relative to ground-truth future ego poses, and consistency across overlapping queries is not evaluated. These gaps limit the evidence for query independence and reliable interaction prediction.

Questions for authors
  1. Could you compare BEP-World with a modular multi-agent predictor followed by spatial querying, using matched inputs and computational budgets? What advantage does joint modeling provide?

  2. Could you provide a direct QAMA ablation or a spatial-coverage-matched baseline to isolate the contribution of query-conditioned prediction?

  3. How do you distinguish trajectory following from interaction modeling on BEP-Pairs? Could you also evaluate query windows without ground-truth future ego poses and measure agreement across overlapping queries?

4 comments
AuthorsOpenReview ↗

[POST 3-1] Reply to Reviewer e9kg (Part 1/3)

We sincerely thank the reviewer for the constructive review. We are encouraged that the reviewer found that separating plan conditioning from spatial querying makes plan comparisons more flexible, addresses a limitation of ego-centric setups, and could be useful for cooperative driving and plan evaluation. In summary (more details in the point-by-point response):

  • Joint vs. modular (W1, Q1). Joint conditional forecasting covers the requested region: Direct reaches 11.66% N-mIoU outside the ego window, where the modular path has no predictions. Our model also supports the modular readout inside the ego window, and combining both (Hybrid) adds about 1 pp. Against multi-agent trajectory predictors with occupancy rendering, BEP-World leads by 6.1–10.2 pp vehicle mIoU.
  • QAMA (W2, Q2). Retrained ablations at matched coverage show that Full QAMA outperforms the combined simplification on all five metrics, by up to 5.10 pp vehicle mIoU.
  • Interaction (Q3). In BEP-Pairs, supplying plans significantly changes the predicted futures of vehicles without supplied plans (95% CI of the gain [1.25, 3.76] pp), which trajectory following cannot do (0%).
  • Fixed queries and consistency (Q3). Plan gains persist in fixed world windows that use no future ego input. Overlapping queries agree on at least 97.5% of shared voxels, and Hybrid makes them agree by construction inside the ego support.

W1 + Q1: What does joint modeling provide over a modular alternative?

Under identical inputs, joint conditional forecasting provides two things that a modular "forecast in the ego frame, then query" pipeline does not.

  • Coverage of the requested region. In S3, Direct reaches 11.66% N-mIoU outside the future ego window, where the modular pipeline has no predictions (0%).
  • The lowest cost for a single region. Direct takes 283 ms per window, versus 291 ms for the modular pipeline.

Modular comparison with matched inputs. We fix the weights, history, plans, and output resolution of the same multi-agent predictor and compare three inference paths. Table R3-1 shows the comparison of them. Warp is the modular alternative: forecast the future ego window, then resample it into the query. Direct forecasts the query window. Hybrid uses Warp inside the ego support and Direct elsewhere.

Table R3-1: S3, same seed-17 checkpoint, 4,117 origins. The metrics averaged over k = 1/3/7, ten non-ego queries, and 1/2/3 s.

Inference pathQ-mIoU ↑N-mIoU ↑V-mIoU ↑
Direct (ours)37.39311.66449.884
Warp (modular: ego forecast → query)37.9800.00050.298
Hybrid (ours: Warp inside ego support + Direct outside)38.38211.66451.013

As the reviewer notes (37.75 vs. 38.56 in the paper), Warp is slightly more accurate inside the ego window, and it is cheaper for many windows since it reuses one forecast. Our model provides this path as well, and Hybrid combines the two to gain about 1 pp Q-mIoU over Direct while keeping Direct's predictions outside the ego window. Direct is the fastest when a few regions are evaluated under many candidate plans, which is the use case that motivates the task.

Other modular references in the paper. CTG++ᴿ and CCDiffᴿ (Table 1c of the paper) are multi-agent trajectory predictors followed by shared occupancy rendering. They use additional inputs (maps and actor histories) and different training, so this comparison is system-level. BEP-World reaches 55.77 V-mIoU in S2 versus 45.58 / 45.59 for these two methods. We additionally render their predicted trajectories into the ten S3 non-ego query windows with the same shared renderer. They reach 43.65 / 43.63 V-mIoU with plan gains of +0.14 / +0.04 pp, compared with 49.77 V-mIoU and +1.12 pp for BEP-World (three-seed mean). Copy-Paste (persistence + plan transport + query readout), a training-free modular reference, reaches 46.64 V-mIoU (+0.12 pp) in S3. Despite their additional inputs, they trail BEP-World by 6.1–10.2 pp V-mIoU.

Coverage-matched modular alternative. Forecasting a larger ego-frame region that contains all queries and then cropping is Warp with a larger window, and it trades compute or resolution for coverage. Such a region needs about 94 m of extent, i.e., about 2.2× our XY tokens at 0.4 m on every request, or about 0.6 m cells at matched compute. At a comparable cost, Hybrid (1.90× Direct for one window) already combines Warp's full-resolution forecast inside the ego support with Direct's predictions outside it. A larger window also has coverage fixed at training time, whereas query conditioning spends a fixed budget at full resolution on whichever region is requested.

AuthorsOpenReview ↗

[POST 3-2] Reply to Reviewer e9kg (Part 2/3)

W2 + Q2: Direct QAMA ablation at matched coverage

We isolate the two capabilities through controlled comparisons. At identical spatial coverage, the retrained QAMA ablation measures the contribution of query-aligned prediction. In BEP-Pairs, changing only the non-ego plan produces measurable responses in vehicles whose futures are never supplied, separating interaction responses from trajectory following (Q3).

As presented in Table R3-2, we have retrained a QAMA ablation with matched inputs, coverage, and recipe. Full QAMA outperforms the combined simplification on all five metrics. Because both models predict the entire query window and are scored on the same voxels, the differences isolate the contribution of the query-aligned design at matched coverage.

Table R3-2: QAMA ablation. Report by percentage.

ModelMulti-frame alignmentMemory attentionUnknown featureS0 mIoU ↑S1 Q-mIoU ↑S1 N-mIoU ↑S2 V-mIoU ↑S3 V-mIoU ↑
Full✓✓✓42.34738.50112.46355.71249.884
Combined simplification×××40.88335.52610.44851.70044.786

In the combined simplification, query-relative alignment and frame encoding apply only to the latest latent at each step. Older history features still enter the temporal backbone in their stored frames, with the backbone's learned temporal-index embeddings but without explicit query-relative pose information. Removing the unknown feature replaces it with zeros.

Full improves over the combined simplification by 2.98 / 2.02 pp S1 full-window / novel-region mIoU and 4.01 / 5.10 pp S2 / S3 vehicle mIoU. The ablation uses seed 17, and these gains are several times the main model's seed SD (0.44 for S1 Q-mIoU and 0.26 / 0.20 for S2 / S3 V-mIoU).

The concerns of interaction responses, dependence on future ego poses, and overlap consistency are addressed in Q3 below.

AuthorsOpenReview ↗

[POST 3-3] Reply to Reviewer e9kg (Part 3/3)

Q3: Trajectory following vs. interaction, fixed queries, and overlap consistency

1. Separating trajectory following from interaction. Each BEP-Pairs pair keeps history and the world-fixed query identical and changes only the controlled vehicles' plans. Responders' futures differ between the two branches and are never supplied. Responder-change F1 scores whether predicted occupancy additions and removals for responders match the simulated changes, with controlled-vehicle regions excluded. Trajectory following alone therefore cannot raise this metric, as confirmed by local edit, which transports controlled content and scores 0%.

For the 60 K = 2 pairs (29 layouts), adding the surrounding plan raises responder F1 from 8.65% to 11.12% (layout-paired 95% CI of the gain: [1.25, 3.76] pp). On 20 pairs (19 layouts) where only the non-ego plan changes and history, ego plan, and query are fixed, Full reaches 5.98% (95% CI of the gain over local edit: [3.44, 8.68] pp), while ego-only and local edit score 0%. These responses come with a stable false-change rate of only 0.90%. The model therefore captures interaction effects that trajectory following cannot.

2. Queries without future ego ground truth. We added two fixed world windows in Table R3-3: F0 at the ego pose at t = 0, and F5 shifted 5 m ahead. Both keep their pose throughout the horizon, and window construction uses only the current pose. In conditions without an ego plan, no input derived from future ego poses is used at inference, which we verified with a voxelwise output-invariance audit.

Table R3-3: Plan availability in fixed windows (averaged over F0/F5, mean ± SD over three seeds, k = 1/3/7 averaged when surrounding plans are supplied).

Ego planSurrounding plansQ-mIoU ↑V-mIoU ↑ΔV (pp) ↑
××50.01 ± 0.8062.42 ± 0.54—
×✓50.24 ± 0.8963.23 ± 0.60+0.82 ± 0.23
✓×48.25 ± 0.6162.70 ± 0.51—
✓✓48.86 ± 0.5663.49 ± 0.52+0.79 ± 0.24

All conditions use the 1,455 of 4,117 origins whose labels cover both windows over the full horizon (median 3-s ego displacement 3.2 m). We score the 89.4%–89.9% of query voxels that lie fully inside the label volume. Future GT poses are used only for label alignment and origin selection. Without any ego plan or future ego pose, surrounding plans still improve V-mIoU by +0.82 pp, as much as with the ego plan, so query definition and prediction do not depend on future ego ground truth.

3. Consistency across overlapping queries. With history and all plans fixed (k = 7), we predict each window independently and compare classes at shared physical locations (three-seed means, 4,117 origins) in Table R3-4 below. The GT rows apply the same comparison to the ground truth rasterized independently in the two windows.

Table R3-4: Overlap consistency (three-seed means over 1/2/3 s).

Query pairMetricMean
Ego ↔ +5 m forwardClass agreement ↑98.86%
Ego ↔ +5 m forwardNon-free disagreement ↓12.77%
Ego ↔ +5 m forwardNon-free disagreement, GT reference20.39%
Ego ↔ 10 non-ego queriesClass agreement ↑97.59%
Ego ↔ 10 non-ego queriesNon-free disagreement ↓23.42%
Ego ↔ 10 non-ego queriesNon-free disagreement, GT reference13.30%

Independent predictions agree on 97.6%–98.9% of shared voxels. Because free space dominates this number, we also report non-free disagreement. For the +5 m pair, predictions are more consistent than the two GT rasterizations (12.77% vs. 20.39%), since the 12.5-voxel shift resamples object boundaries. Across the ten queries, non-free disagreement is 23.42%, about 10 pp above the GT reference, mainly from rotated queries (32.3%–36.8% vs. 16.0% for translated ones). When agreement is required, our model already provides it: Hybrid reads every query inside the ego support from one shared forecast, so overlapping queries agree by construction there, while improving accuracy by about 1 pp. Beyond ego support, a training-time agreement loss at shared physical points is direct future work.

We hope these results address the reviewer's concerns. We would be grateful if the reviewer would consider them in the final assessment, and we are happy to provide further details.

AuthorsOpenReview ↗

Discussion Invitation

Dear e9kg, thanks for taking the time and effort to help us improve this work.

We understand that too much detail can make a reply harder to read, so we put a summary at the beginning of Part 1. You can skim it first. If anything is still a concern, you can find the details in our point-by-point replies below. We included these details because we hoped to address your concerns as fully as possible.

A brief summary

  1. With the same inputs, Direct reached 11.66% mIoU in regions beyond the ego window, where Warp has no predictions. Combining the two improved full-window mIoU by about one point over Direct.

  2. At the same spatial coverage, full QAMA outperformed the retrained simplified version by 5.10 points on S3 vehicle mIoU.

  3. In 20 BEP-Pairs cases where only one neighboring vehicle’s plan changed, BEP-World captured some of the resulting changes in other vehicles’ occupancy (5.98% responder F1, versus 0% for rigid transport).

We’d be happy to hear your thoughts or discuss any remaining concerns.

Reviewer_ZjC1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of learning an occupancy world model for autonomous driving, which predicts how the 3D occupancy around the car evolves over the next few seconds. The authors identify a strong coupling between the output future occupancy and the ego plan in two ways: (1) the model conditions only on the ego's plan and (2) the output is always defined centered at the ego's future pose. This paper proposes to decouple both by conditioning on any subset of agents (ego included but optional) and querying output in a shifted and rotated window (no longer centered at ego). This allows one to study the model under different settings, including asking what-if questions in the context of occupancy world models (previously explored in the context of trajectory forecasting). The proposed method outperforms prior methods on OpenOccupancy, but against a copy-paste baseline (copy the current scene with the planned cars moved), the results are mixed.

Critical strengths

The problem is well motivated. Prior occupancy world models condition only on ego's plan and predict only centered around ego, which limits what these models can answer. Conditioning on other agents' plans and choosing where to query enables questions like "what happens at the merge ahead if the truck cuts in?".

Critical weaknesses
  1. The results appear to use ground-truth information, including both ground-truth history occupancy as input (prior works report camera-input settings, e.g., OccWorld and COME) and true future plans of other agents.
  2. The results against the copy-paste baseline are mixed. BEP-World underperforms Copy-Paste on full-scene mIoU in Table 1 under S1 - S3 (38.90 vs 39.79 for S1, 42.00 vs 42.58 for S2, 37.75 vs 39.12 for S3). As the authors also note in Appendix C.1, "the mean S0 advantage over Copy-Paste is smaller than the downstream seed variation".
Questions for authors
  1. Is it possible to compare to Copy-Paste on only the dynamic voxels? By design, does BEP-World outperform Copy-Paste there?
  2. The baselines in Table 1 S0 are trained with different grids. How are they adapted? Are they re-trained? Or are their results resampled? If so, how are they resampled?
  3. How is Copy-Paste in this work different from the one in OccWorld?
  4. How sensitive is the proposed model to noise in conditioning? Have the authors try replacing ground-truth signals in conditioning with estimation?
  5. Since cooperative driving is used to motivate the work, I wonder if the authors have tried evaluating BEP-World for selecting candidate plans. I wonder if it could help drive down metrics such as collision rates when tested in closed-loop simulation.
6 comments
AuthorsOpenReview ↗

[POST 2-1] Reply to Reviewer ZjC1 (Part 1/5)

We sincerely thank the reviewer for the thoughtful review. We are glad that the reviewer found the problem well motivated: prior occupancy world models condition only on the ego plan and predict only around ego, whereas conditioning on other agents and choosing where to query makes questions such as "what happens at the merge ahead if the truck cuts in?" answerable. We also appreciate the note that BEP-World outperforms prior methods on OpenOccupancy. In summary (more details in the point-by-point response):

  • Copy-Paste (W2, Q1). BEP-World outperforms Copy-Paste on dynamic classes in all four settings and for every seed, which is the content that forecasting and plan conditioning target. Copy-Paste's static advantage comes from copying the current GT frame exactly.
  • Ground-truth conditioning (W1, Q4). With perception-predicted or corrupted history from the OccStress benchmark, BEP-World leads Copy-Paste on every metric, and its Q-mIoU margin grows from 0.75 pp with GT history to 3.1–4.2 pp. Plan gains persist under plan errors up to 0.5 m RMS, and naive constant-velocity plans cost only 0.16 pp relative to ego-only conditioning.
  • Baselines (Q2, Q3). All learned baselines were retrained on our grid and predicted on it directly. Our Copy-Paste adds plan transport and query readout to OccWorld's version, which makes it a strong training-free reference.
  • Plan selection (Q5). The decoupled plan and query interface is what plan selection requires, and BEP-Pairs already evaluates plan changes under a fixed query.

W1: Ground-truth history and plans

Historical occupancy. In S0, OccWorld, COME, and BEP-World all use the same GT historical occupancy, and OccWorld and COME were retrained on our grid. Their camera-input scores were not used. GT history isolates forecasting from perception errors, a setting both original works also report. We additionally replace GT history with perception-predicted and corrupted occupancy from the OccStress benchmark, where BEP-World's advantage over Copy-Paste becomes larger (Table R2-4).

Plans. We predict the future scene in a query region, given plans for selected vehicles, which may be specified by a user or shared by cooperating vehicles. On logged data, recorded trajectories instantiate these plans so that predictions can be scored against the observed future. Neither the futures of vehicles without supplied plans nor future occupancy is ever an input. Within each setting, all compared methods receive identical conditioning. S0/S1 use no surrounding plans. In S2, all methods share the same selected vehicles, XY waypoints, timestamps, validity masks, and current geometry. In S3, Copy-Paste receives the same plans as BEP-World.

AuthorsOpenReview ↗

[POST 2-2] Reply to Reviewer ZjC1 (Part 2/5)

W2 + Q1: Comparison with Copy-Paste on dynamic content

Thanks. Your suggestion to examine dynamic content helps to proof what the model learns about motion. BEP-World outperforms Copy-Paste on dynamic classes in all four settings. The full-scene gap in S1–S3 comes from static classes, where Copy-Paste copies the current GT frame exactly. We split the 16 semantic classes into dynamic and static groups and score each group on the full window. The dynamic group is defined by semantic class and therefore also includes parked vehicles.

Table R2-1: Dynamic vs. static classes (full-window 3D mIoU, %). BEP-World: mean ± SD over three seeds.

SettingMethodDynamic ↑Static ↑All ↑
S0BEP-World34.83 ± 0.6651.43 ± 0.9743.13 ± 0.81
S0Copy-Paste32.8452.8642.85
S1BEP-World31.14 ± 0.4046.66 ± 0.4838.90 ± 0.44
S1Copy-Paste30.5649.0339.79
S2BEP-World34.93 ± 0.6149.06 ± 0.9742.00 ± 0.79
S2Copy-Paste32.3352.8342.58
S3BEP-World30.91 ± 0.3844.58 ± 0.4837.75 ± 0.43
S3Copy-Paste29.6248.6239.12

Dynamic-class gains are +1.98, +0.58, +2.61, and +1.30 pp in S0–S3 (from unrounded scores) and are positive for every seed. These are the classes whose future depends on motion and plans, and they are where a forecasting model must add value over persistence. Each group contains eight classes, so overall mIoU is the mean of the two group scores.

S0 and seed variation. We agree with the reviewer's reading of Appendix C.1: the overall S0 margin over Copy-Paste (0.28 pp) is within seed variation (SD 0.81 pp). The dynamic-class margin (+1.98 pp) is positive for all three seeds, so BEP-World's S0 advantage lies in dynamic content, where forecasting matters.

The static gap is a representation trade-off. The gap is not uniform. In S3, BEP-World is better than Copy-Paste on driveable surface (+1.60 pp) and barrier (+0.37 pp), and the gap concentrates on small or intricate structures: traffic cones (−9.39 pp), manmade structures (−8.71 pp), vegetation (−5.41 pp), and terrain (−5.12 pp). Copy-Paste copies the current GT frame exactly, which is an advantage for structures that persist over 3 s. BEP-World instead encodes history into a compact tri-plane, which is what allows it to decode arbitrary query regions and condition on many agents at 283 ms per window. To quantify this trade-off, we encode the current GT frame (t = 0) with each Full model's final tokenizer and decode it without any forecasting in Table R2-2 below.

Table R2-2: Reconstruction of the current frame at t = 0 (mIoU, %). Full tokenizer: mean ± SD over the three seeds.

RepresentationDynamic ↑Static ↑Traffic cone ↑Manmade ↑Vegetation ↑Terrain ↑
Copy-Paste (identity)100.00100.00100.00100.00100.00100.00
Full tokenizer, encode + decode55.00 ± 2.6566.01 ± 0.7035.19 ± 1.6459.67 ± 2.2861.08 ± 2.3270.99 ± 0.97

Copy-Paste starts every forecast from an exact copy, whereas our tokenizer reconstructs static classes at 66.01 mIoU and traffic cones at 35.19 before any forecasting. The static gap is therefore strongly tied to tokenizer fidelity. The dynamic classes give the clearest evidence of forecasting ability: starting from a reconstruction of only 55.00, BEP-World still forecasts them better than Copy-Paste's exact copy, so its dynamic advantage comes from modeling motion. Tokenizer fidelity is orthogonal to our contributions, plan conditioning and query-aligned forecasting, and a higher-fidelity tokenizer can be used in the same framework. The static advantage of Copy-Paste also depends on exact GT input, and with perception-predicted history BEP-World leads on every metric in Table R2-4.

AuthorsOpenReview ↗

[POST 2-3] Reply to Reviewer ZjC1 (Part 3/5)

Q2: Baseline adaptation and retraining

Thanks for your question. We clarify that all five learned S0 baselines were retrained on our grid and predicted directly on it. No cross-grid resampling is used for evaluation. All methods use 160 × 160 × 20 voxels at 0.4 m, covering x, y ∈ [−32, 32) m and z ∈ [−3, 5) m in each frame's ego coordinates.

Table R2-3: Adaptation and retraining of the S0 baselines.

MethodGrid adaptationTraining
OccWorldVQ-VAE spatial hierarchy, vertical dimension, predictor token gridVQ-VAE and predictor from scratch
DOMEVAE geometry and diffusion latent gridFrozen VAE trained for COME on the shared grid, then the DOME predictor
COMEVAE, scene predictor, coordinate-alignment rangeVAE, base predictor, scene UNet, and ControlNet on the shared grid
GenieDriveTri-plane grid and occupancy decoderTokenizer from scratch, predictor with frozen tokenizer, then end-to-end
I²-WorldTokenizer dimensions, spatial token grid, spatial loss-weight mapsTokenizer from scratch, re-exported tokens, then world model

Each method keeps its own training stages and forecasting mechanism as shown in Table R2-3. Because the grid, label source (OpenOccupancy instead of Occ3D), and 3-s horizon differ from the original papers, absolute scores are not comparable to the numbers reported there.

Inputs and GT are built from OpenOccupancy's 0.2m annotations. Occupied voxel centers are transformed into each frame's ego coordinates, cropped, and binned into 0.4m voxels by majority class, with ties resolved by the smaller class ID. All methods share the class mapping, ego-body template, the same 4,519 origins across 150 validation scenes, and the same confusion-matrix scoring.

Q3: Difference from OccWorld's Copy&Paste

OccWorld's Copy&Paste copies the current GT occupancy as every future frame. Ours additionally transports planned vehicles and reads out arbitrary query windows, making it a stronger, training-free reference.

  • Plans: selected vehicles' current voxels are rigidly moved along the supplied XY plans (heading from waypoints, current size and height retained), and vacated voxels are cleared.
  • Remaining scene: vehicles without plans and all other content stay at their world positions.
  • Queries: the resulting world-frame scene is read directly into the requested window, without an intermediate crop to the future ego window. Unsupported voxels are labeled free and scored.

S0/S1 use only the ego plan, and S2/S3 add the selected surrounding plans. This baseline measures how far observation persistence plus geometric transport can go, which is why it outperforms several learned forecasters. BEP-World surpasses it on dynamic content in every setting.

Reviewer_LsAVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines a "queryable multi-agent 4D occupancy world model". Given four past semantic occupancy grids, it forecasts six future grids inside a 64 × 64 × 8 m window whose pose the user chooses at each future step (the "query"). The forecast is conditioned on box trajectories ("plans") supplied for any subset of up to eight vehicles, with ego as an optional member of that set.

Critical strengths
  • It's a clearly defined task. the Plans say how selected vehicles move and queries say where the future is reported, and ego enters through the same plan interface as any other vehicle.
  • Compared w/ exisitng works, forecasts beyond the future ego window is interesting, and plans are used mainly through the spatial field.
  • Reproducibility looks good w/ an anonymous code link, the full three-stage recipe, the field channels and the metric definitions are given.
Critical weaknesses
  • The motivating use case is never run on real data. The abstract, the introduction and Figure 1 motivate comparing plans in one fixed road region, "not in a window that changes with each ego plan". But all eleven OpenOccupancy queries are offsets of the contemporaneous future ego frame.
  • The real-data evidence shows forecasting in shifted ego windows plus gains from logged surrounding-vehicle futures. It does not show the plan comparison in a fixed region that defines the task.
  • The query-frame design is not shown to beat simpler readouts, and cross-query consistency is not measured.
  • The plan-conditioning gains are small, pooled over all vehicles, and partly matched by geometric transport
Questions for authors
  • Can you report S3, and the ego-plan availability study, with query poses fixed in the t = 0 frame? Please use origins where the window stays inside the label volume over the horizon, and say how many origins qualify.
  • What do a hybrid readout (Warp inside the ego-window support, Direct outside it) and a matched-budget ego-frame forecast over a window that contains all evaluated queries achieve on S1/S3? Can you add a trained ablation of Query-Aligned Memory Aggregation, for example latest-frame alignment only, without distance-biased memory attention or the unknown feature?
  • For fixed history and plans, what fraction of shared physical voxels receive the same class in two overlapping queries (for example the ego window and +5 m forward), per horizon?
5 comments
AuthorsOpenReview ↗

[POST 1-1] Reply to Reviewer LsAV (Part 1/4)

We sincerely thank the reviewer for the careful reading and the valuable suggestions. We are encouraged that the reviewer found the task clearly defined: plans specify how selected vehicles move, queries specify where the future is reported, and ego enters through the same plan interface as any other vehicle. In summary (more details in the point-by-point response):

  • Fixed road regions (W1, W2, Q1). Plan conditioning does not rely on future ego information. In world-fixed windows without any future ego input, surrounding-vehicle plans raise vehicle mIoU by +0.8 pp, matching the gain under moving queries, and BEP-World outperforms Copy-Paste on vehicle mIoU in every window. Plan comparison under a shared fixed query is evaluated directly in BEP-Pairs.
  • Readouts (W3, Q2). Our model supports both readouts. Hybrid adds about 1 pp, while every prediction outside the ego window still comes from query-conditioned Direct, which no ego-window readout can provide. A larger ego-centered window would need about 2.2× the tokens on every request. It is exactly the cost that query conditioning avoids.
  • QAMA (Q2). Full QAMA outperforms the requested combined simplification on all five metrics, by up to 5.10 pp vehicle mIoU.
  • Plan gains, interaction, and consistency (W4, Q3). On the vehicles whose plans are supplied, gains are more than twice the pooled value, and they avoid the large losses that rigid transport suffers on trucks, trailers, and construction vehicles. In BEP-Pairs, plans significantly change the predicted futures of vehicles without supplied plans, but transport cannot do. For overlapping queries, Hybrid makes predictions agree by construction inside ego support.

W1 + W2 + Q1: Forecasting and plan comparison in a fixed road region

That's really a genius idea that using fixed region to help us control variable. Real driving logs contain a single factual future per origin, so comparing different plans in the same region cannot be scored on real data. We test the two halves of the motivating use case separately: (i) forecasting in a fixed world window on OpenOccupancy, with and without surrounding-vehicle and ego plans (Table R1-1 below), and (ii) plan comparison under a shared world-fixed query in BEP-Pairs (please kindly refer to Sec. 4.3, App. D.1). In BEP-Pairs, history and query are identical within each pair and only the plans change.

Setup. Your suggestion to anchor the queries at t = 0 gives us a concrete way to test fixed-region forecasting. And F5 is shifted 5 m forward along the ego heading at t = 0.

Origins and label support. We require the window's footprint to stay inside the OpenOccupancy label volume with a 2 m margin at all 6 future steps. 1,455 of 4,117 origins qualify for F0 and 2,106 for F5, and all conditions use their intersection of 1,455 origins (35.34%). Because the window must stay inside the label volume while the ego moves. We score voxels fully inside the label volume, 89.4%–89.9% of each fixed query grid.

Table R1-1: Results of S3 with fixed windows. Ego and surrounding plans, equally averaged over k = 1/3/7. BEP-World: mean ± SD over 3 seeds. Copy-Paste receives the same plans. The moving-query reference is recomputed on the same 1,455 origins with the same support rule.

Query settingMethodQ-mIoU ↑N-mIoU ↑V-mIoU ↑ΔV from surrounding plans (pp) ↑
Original moving queries (means over 10 windows)BEP-World42.01 ± 0.3810.58 ± 0.0255.01 ± 0.31+0.90 ± 0.19
Copy-Paste46.275.6453.43−1.12
Fixed window F0BEP-World51.01 ± 0.4141.78 ± 0.2365.93 ± 0.42+0.77 ± 0.25
Copy-Paste57.5349.3962.84−1.70
Fixed window F5BEP-World46.72 ± 0.7423.16 ± 0.4161.06 ± 0.63+0.81 ± 0.23
Copy-Paste48.6421.1054.37−1.12

Surrounding-vehicle plans improve BEP-World's V-mIoU by +0.77 / +0.81 pp in F0 / F5, comparable to the +0.90 pp obtained with moving queries. Plan utilization does not depend on the query moving with ego. The comparison with Copy-Paste shows why learned plan conditioning matters: the same plans lower Copy-Paste's V-mIoU by 1.1–1.7 pp but raise ours, and BEP-World leads on V-mIoU by +1.6 / +3.1 / +6.7 pp. Copy-Paste's higher Q-mIoU here reflects static persistence. On the OccStress robustness benchmark, with history predicted by the camera-based STCOcc and SDGOcc networks (S3, query 5 m ahead), BEP-World leads Copy-Paste on every metric, by 3.1–4.2 pp Q-mIoU and 3.4–5.2 pp V-mIoU.

AuthorsOpenReview ↗

[POST 1-2] Reply to Reviewer LsAV (Part 2/4)

W1 + W2 + Q1 (continued): Ego-plan availability in fixed windows

Table R1-2: Ego-plan availability with fixed windows. Equally averaged over F0/F5. Mean ± SD over three seeds. Conditions with surrounding plans average k = 1/3/7.

Ego planSurrounding plansQ-mIoU ↑V-mIoU ↑ΔV (pp) ↑
××50.01 ± 0.8062.42 ± 0.54—
×✓50.24 ± 0.8963.23 ± 0.60+0.82 ± 0.23
✓×48.25 ± 0.6162.70 ± 0.51—
✓✓48.86 ± 0.5663.49 ± 0.52+0.79 ± 0.24

Without an ego plan, surrounding plans still improve V-mIoU by +0.82 pp, as much as the +0.79 pp obtained with it. The ego plan is therefore not a required input for fixed-window forecasting, and the query does not rely on future ego ground truth.

The ego plan is an optional input that shifts accuracy toward dynamic content. In fixed windows it raises dynamic-class mIoU by +0.66 ± 0.16 pp and V-mIoU by about 0.3 pp, while ground surfaces decrease (driveable surface −7.6, terrain −7.1, sidewalk −6.3 pp), which lowers Q-mIoU by 1.76 pp. This ground-class change is about 2.5× smaller under the moving queries used in training (−2.7 vs. −6.7 pp). Because the interface makes every plan optional, a user can omit the ego plan in a fixed region and keep the full surrounding-plan gain (+0.82 pp).

The comparison half. In BEP-Pairs, with history and world-fixed query held constant, supplying the second vehicle's plan raises responder-change F1 from 8.65% to 11.12%, whereas local edit scores 0% (details in W4).

We will state this two-part evidence explicitly in the revised paper.

W3 (simpler readouts) + Q2: Hybrid readout and larger-window forecasting

Our model supports both readouts, and only query-conditioned Direct reaches regions outside the ego window. Thanks to your comments, we have conducted a comparison between different modes in Table R1-3 below. Hybrid uses Warp inside ego support and Direct outside it, improves full-window accuracy by about 1 pp over Direct. All of its predictions outside the ego window still come from Direct, and Direct is cheaper when only a few regions are requested. Hybrid is an additional readout that our query-conditioned model enables, not a substitute for it.

Table R1-3: Comparison between Direct, Warp, and Hybrid readout modes (same seed-17 checkpoint, history, and plans). S1 Q/V scores average 11 queries. S3 scores average k = 1/3/7 and 10 non-ego queries. N covers newly queried regions of non-ego queries. All metrics average 1/2/3 s.

SettingReadoutQ-mIoU ↑N-mIoU ↑V-mIoU ↑
S1Direct38.50112.46349.200
S1Warp39.1770.00049.760
S1Hybrid39.64912.46350.348
S3Direct37.39311.66449.884
S3Warp37.9800.00050.298
S3Hybrid38.38211.66451.013

Relative to Direct, Hybrid improves Q-mIoU by 1.149 / 0.989 pp and V-mIoU by 1.149 / 1.129 pp in S1 / S3. Its predictions in newly queried regions are identical to Direct's at every voxel.

Table R1-4: Measured latency (ms). One A100, FP32, batch 1, PyTorch MSDA backend. Median over 32 origins × 3 repeats. Timing includes history encoding, six-step forecasting and decoding, GPU-to-CPU transfer, and Warp/merging, and excludes disk I/O, model loading, and input preparation.

RequestDirect ↓Warp ↓Hybrid ↓Hybrid / Direct
1 non-ego window283.4290.8539.81.90×
10 non-ego windows2480.3353.42867.81.16×

Warp amortizes one ego forecast across many windows, while Hybrid costs 1.90× Direct for a single window and 1.16× for ten. In the motivating use case, where a few regions are evaluated under many candidate plans. Every method needs one rollout per plan, so the single-window cost is the relevant one, and Direct is the fastest.

Larger-window forecasting followed by cropping. This alternative is Warp with a larger ego window, and it trades compute or resolution for coverage. The 11 evaluated queries reach 47 m ahead of the future ego frame and 45 m laterally after the ±45° rotations, so an ego-centered window containing all of them needs about 94 m of extent. At our 0.4 m resolution this is about 2.2× the XY tokens of our 160 × 160 grid on every forward pass. Keeping 160 × 160 instead coarsens the cells to about 0.6 m. It reduces the resolution available for small objects such as pedestrians and traffic cones. At a comparable cost, our model already provides what such a window aims for: Hybrid costs 1.90× Direct for a single window, keeps Warp's full-resolution forecast inside the ego support, adds Direct's predictions outside it, and improves Q-mIoU by about 1 pp (Table R1-3). The coverage of a larger window is also fixed at training time, so a region beyond it. Query conditioning has no such limit, because it spends a fixed budget at full resolution on whichever region is requested.

AuthorsOpenReview ↗

[POST 1-3] Reply to Reviewer LsAV (Part 3/4)

Q2 (continued): Trained ablation of Query-Aligned Memory Aggregation

That is a good point. Based on your suggestions, we have added a trained ablation of QAMA as shown in Table R1-5. Full QAMA outperforms the requested combined simplification (latest-frame alignment only, no distance-biased memory attention, no learned unknown feature) on all five metrics, by 2.98 pp S1 Q-mIoU and 5.10 pp S3 V-mIoU. The simplified variant reuses Full's tokenizer.

Table R1-5: QAMA ablation (seed 17, %). S1 Q-mIoU averages 11 queries. N-mIoU covers newly queried regions of the 10 non-ego queries. S2/S3 V-mIoU average k = 1/3/7.

ModelMulti-frame alignmentMemory attentionUnknown featureS0 mIoU ↑S1 Q-mIoU ↑S1 N-mIoU ↑S2 V-mIoU ↑S3 V-mIoU ↑
Full✓✓✓42.34738.50112.46355.71249.884
Combined simplification×××40.88335.52610.44851.70044.786

In the combined simplification, query-relative alignment and frame encoding apply only to the latest latent at each step. Older history features still enter the temporal backbone in their stored frames, with the backbone's learned temporal-index embeddings but without explicit query-relative pose information. Removing the unknown feature replaces it with zeros.

Compared with the combined simplification, S0-S3 are both improved in full BEP-World. The query-aligned design contributes substantially to both spatial querying and multi-agent forecasting.

W3 (consistency) + Q3: Agreement across overlapping queries

Independent predictions of overlapping queries agree on 97.5%–98.9% of shared voxels in Table R1-6, and for nearby queries they are more consistent than independently rasterized ground truth. We fix the history, ego plan, and up to k=7 surrounding plans, forecast each query window independently, and compare classes at shared physical locations.

Table R1-6: Overlap consistency by horizon. The class agreement covers all shared voxels, including free space.

Query pairMetric1 s2 s3 sMean
ego ↔ +5 m forwardClass agreement ↑98.83%98.87%98.89%98.86%
ego ↔ +5 m forwardNon-free disagreement ↓12.47%12.66%13.19%12.77%
ego ↔ +5 m forwardNon-free disagreement, GT reference20.31%20.38%20.49%20.39%
ego ↔ 10 non-ego queriesClass agreement ↑97.68%97.58%97.50%97.59%
ego ↔ 10 non-ego queriesNon-free disagreement ↓21.52%23.31%25.43%23.42%
ego ↔ 10 non-ego queriesNon-free disagreement, GT reference13.24%13.29%13.37%13.30%

Because free space dominates class agreement, we also report non-free disagreement. For the +5 m pair, predictions disagree less than the two GT rasterizations (12.77% vs. 20.39%), since a 12.5-voxel shift resamples object boundaries. Across the ten queries, non-free disagreement is 23.42%, about 10 pp above the GT reference, and comes mainly from rotated queries (32.3% at ±30° and 36.8% at ±45°, vs. 16.0% for translated queries). When agreement is required, our model already provides it: Hybrid reads every query inside the ego support from one shared forecast, so overlapping queries agree by construction there, and it also improves accuracy by about 1 pp (Table R1-3). A future extension is a training-time agreement loss at shared physical points, with full GT supervision retrained for each window.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary and Main Contributions

We thank all three reviewers and the Area Chair for their time and effort. During the rebuttal, we ran additional experiments and clarified to each point.

Discussion Summary:

  • Reviewer LsAV found our task interesting, particularly the ability to forecast beyond the future ego window. A key question was whether surrounding-vehicle plans still help when the query window stays fixed. Our real-data experiments show that these plans improve vehicle forecasts in a fixed road region even without an ego plan. We also tested the suggested hybrid readout and measured agreement across overlapping queries. The fixed windows experiment helps us usefully to proof our main contributions.

  • Reviewer ZjC1 found the problem well motivated. The main concerns were how BEP-World compares with Copy-Paste and how it performs with imperfect inputs. BEP-World outperformed Copy-Paste on dynamic classes across all four settings, while Copy-Paste performed better on static classes. With forecasting chains testing of OccStress, BEP-World also achieved higher full-window metrics. These results answer the main concerns of reviewer and give more positive evidence for BEP-World.

  • Reviewer e9kg noted that our formulation makes plan comparisons more flexible. The review called for separate tests of query conditioning and interaction modeling. At the same spatial coverage, full QAMA outperformed the retrained simplified variant. In paired simulations where only one vehicle’s plan changed, BEP-World captured some of the resulting changes in other vehicles’ occupancy, while rigid transport left their predictions unchanged.

Our work makes three Main Contributions:

  1. We identify two forms of ego privilege in occupancy forecasting. Prior models condition on the ego vehicle’s plan and tie the prediction window to its future pose.
  2. We formulate queryable multi-agent 4D occupancy forecasting. The task separates whose plans are supplied from where predictions are made. It asks, “Given plans for selected vehicles, what will happen in the region of interest?”
  3. We introduce BEP-World as a unified solution. A single architecture supports ego-conditioned forecasting, spatial querying, multi-agent plan conditioning, and their combination.

We will incorporate these experiments and clearer analysis into the final version. We thank the reviewers again for their suggestions and hope these new results clarify what BEP-World contributes. We also hope BEP-World, as an initial exploration of forecasting Beyond Ego Privilege, inspires further work on how different vehicles’ plans shape a shared road scene.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.