- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BEP-World, a 4D occupancy model that separates vehicle-plan conditioning from spatial querying. Query-Aligned Memory Aggregation (QAMA) aligns historical scene features with requested query coordinates, while Grounded Plan Fusion (GPF) incorporates plans for ego and surrounding vehicles. Experiments on OpenOccupancy cover ego forecasting, spatial querying, multi-agent conditioning, and their combination. The authors also introduce BEP-Pairs, a CARLA-based paired-intervention dataset. The results show that joint plan conditioning and spatial querying are feasible, although the evidence for predicting responses of unconditioned vehicles is limited.
Critical strengths
- The formulation makes plan comparisons more flexible.
The paper separates which vehicle plans are provided from where future occupancy is queried. This lets ego and surrounding vehicles share a common conditioning interface while evaluating different plans over the same spatial region. The formulation addresses a limitation of ego-centric setups and could be useful for cooperative driving and plan evaluation.
- The evaluation includes targeted interventions and informative ablations.
The S0–S3 protocol separates ego forecasting, spatial querying, multi-agent conditioning, and their combination. Matched-checkpoint comparisons and per-seed results help assess the contribution of additional vehicle plans. BEP-Pairs also distinguishes changes to controlled vehicles from responses of unconditioned vehicles. The GPF ablations favor spatial fields over agent tokens alone, supporting the use of explicit geometric correspondence.
Critical weaknesses
- The paper does not establish why joint modeling is preferable to a modular alternative.
Prior work has explored multi-agent conditioning and spatial querying. BEP-World combines them in one interface, but the paper does not compare it with a modular multi-agent predictor followed by spatial querying. Without a comparison using matched inputs and computational budgets, it is difficult to tell whether the joint formulation offers a distinct modeling benefit or mainly combines existing capabilities.
- The experiments do not fully isolate the claimed query and interaction capabilities.
The Query-versus-Warp comparison favors direct querying in novel regions, but Warp cannot predict outside its original ego-centered window. Under S3, direct querying also has lower full-window mIoU than Warp (37.75 versus 38.56), and there is no direct QAMA ablation or spatial-coverage-matched baseline. On BEP-Pairs, the use of factual future trajectories makes it difficult to separate interaction modeling from trajectory following: controlled IoU improves from 48.90 to 60.39, but responder-change F1 rises only from 8.65 to 11.12, with 10.03% recall. Some query windows are also defined relative to ground-truth future ego poses, and consistency across overlapping queries is not evaluated. These gaps limit the evidence for query independence and reliable interaction prediction.
Questions for authors
-
Could you compare BEP-World with a modular multi-agent predictor followed by spatial querying, using matched inputs and computational budgets? What advantage does joint modeling provide?
-
Could you provide a direct QAMA ablation or a spatial-coverage-matched baseline to isolate the contribution of query-conditioned prediction?
-
How do you distinguish trajectory following from interaction modeling on BEP-Pairs? Could you also evaluate query windows without ground-truth future ego poses and measure agreement across overlapping queries?
4 comments
[POST 3-2] Reply to Reviewer e9kg (Part 2/3)
W2 + Q2: Direct QAMA ablation at matched coverage
We isolate the two capabilities through controlled comparisons. At identical spatial coverage, the retrained QAMA ablation measures the contribution of query-aligned prediction. In BEP-Pairs, changing only the non-ego plan produces measurable responses in vehicles whose futures are never supplied, separating interaction responses from trajectory following (Q3).
As presented in Table R3-2, we have retrained a QAMA ablation with matched inputs, coverage, and recipe. Full QAMA outperforms the combined simplification on all five metrics. Because both models predict the entire query window and are scored on the same voxels, the differences isolate the contribution of the query-aligned design at matched coverage.
Table R3-2: QAMA ablation. Report by percentage.
| Model | Multi-frame alignment | Memory attention | Unknown feature | S0 mIoU ↑ | S1 Q-mIoU ↑ | S1 N-mIoU ↑ | S2 V-mIoU ↑ | S3 V-mIoU ↑ |
|---|---|---|---|---|---|---|---|---|
| Full | ✓ | ✓ | ✓ | 42.347 | 38.501 | 12.463 | 55.712 | 49.884 |
| Combined simplification | × | × | × | 40.883 | 35.526 | 10.448 | 51.700 | 44.786 |
In the combined simplification, query-relative alignment and frame encoding apply only to the latest latent at each step. Older history features still enter the temporal backbone in their stored frames, with the backbone's learned temporal-index embeddings but without explicit query-relative pose information. Removing the unknown feature replaces it with zeros.
Full improves over the combined simplification by 2.98 / 2.02 pp S1 full-window / novel-region mIoU and 4.01 / 5.10 pp S2 / S3 vehicle mIoU. The ablation uses seed 17, and these gains are several times the main model's seed SD (0.44 for S1 Q-mIoU and 0.26 / 0.20 for S2 / S3 V-mIoU).
The concerns of interaction responses, dependence on future ego poses, and overlap consistency are addressed in Q3 below.
[POST 3-3] Reply to Reviewer e9kg (Part 3/3)
Q3: Trajectory following vs. interaction, fixed queries, and overlap consistency
1. Separating trajectory following from interaction. Each BEP-Pairs pair keeps history and the world-fixed query identical and changes only the controlled vehicles' plans. Responders' futures differ between the two branches and are never supplied. Responder-change F1 scores whether predicted occupancy additions and removals for responders match the simulated changes, with controlled-vehicle regions excluded. Trajectory following alone therefore cannot raise this metric, as confirmed by local edit, which transports controlled content and scores 0%.
For the 60 K = 2 pairs (29 layouts), adding the surrounding plan raises responder F1 from 8.65% to 11.12% (layout-paired 95% CI of the gain: [1.25, 3.76] pp). On 20 pairs (19 layouts) where only the non-ego plan changes and history, ego plan, and query are fixed, Full reaches 5.98% (95% CI of the gain over local edit: [3.44, 8.68] pp), while ego-only and local edit score 0%. These responses come with a stable false-change rate of only 0.90%. The model therefore captures interaction effects that trajectory following cannot.
2. Queries without future ego ground truth. We added two fixed world windows in Table R3-3: F0 at the ego pose at t = 0, and F5 shifted 5 m ahead. Both keep their pose throughout the horizon, and window construction uses only the current pose. In conditions without an ego plan, no input derived from future ego poses is used at inference, which we verified with a voxelwise output-invariance audit.
Table R3-3: Plan availability in fixed windows (averaged over F0/F5, mean ± SD over three seeds, k = 1/3/7 averaged when surrounding plans are supplied).
| Ego plan | Surrounding plans | Q-mIoU ↑ | V-mIoU ↑ | ΔV (pp) ↑ |
|---|---|---|---|---|
| × | × | 50.01 ± 0.80 | 62.42 ± 0.54 | — |
| × | ✓ | 50.24 ± 0.89 | 63.23 ± 0.60 | +0.82 ± 0.23 |
| ✓ | × | 48.25 ± 0.61 | 62.70 ± 0.51 | — |
| ✓ | ✓ | 48.86 ± 0.56 | 63.49 ± 0.52 | +0.79 ± 0.24 |
All conditions use the 1,455 of 4,117 origins whose labels cover both windows over the full horizon (median 3-s ego displacement 3.2 m). We score the 89.4%–89.9% of query voxels that lie fully inside the label volume. Future GT poses are used only for label alignment and origin selection. Without any ego plan or future ego pose, surrounding plans still improve V-mIoU by +0.82 pp, as much as with the ego plan, so query definition and prediction do not depend on future ego ground truth.
3. Consistency across overlapping queries. With history and all plans fixed (k = 7), we predict each window independently and compare classes at shared physical locations (three-seed means, 4,117 origins) in Table R3-4 below. The GT rows apply the same comparison to the ground truth rasterized independently in the two windows.
Table R3-4: Overlap consistency (three-seed means over 1/2/3 s).
| Query pair | Metric | Mean |
|---|---|---|
| Ego ↔ +5 m forward | Class agreement ↑ | 98.86% |
| Ego ↔ +5 m forward | Non-free disagreement ↓ | 12.77% |
| Ego ↔ +5 m forward | Non-free disagreement, GT reference | 20.39% |
| Ego ↔ 10 non-ego queries | Class agreement ↑ | 97.59% |
| Ego ↔ 10 non-ego queries | Non-free disagreement ↓ | 23.42% |
| Ego ↔ 10 non-ego queries | Non-free disagreement, GT reference | 13.30% |
Independent predictions agree on 97.6%–98.9% of shared voxels. Because free space dominates this number, we also report non-free disagreement. For the +5 m pair, predictions are more consistent than the two GT rasterizations (12.77% vs. 20.39%), since the 12.5-voxel shift resamples object boundaries. Across the ten queries, non-free disagreement is 23.42%, about 10 pp above the GT reference, mainly from rotated queries (32.3%–36.8% vs. 16.0% for translated ones). When agreement is required, our model already provides it: Hybrid reads every query inside the ego support from one shared forecast, so overlapping queries agree by construction there, while improving accuracy by about 1 pp. Beyond ego support, a training-time agreement loss at shared physical points is direct future work.
We hope these results address the reviewer's concerns. We would be grateful if the reviewer would consider them in the final assessment, and we are happy to provide further details.
Discussion Invitation
Dear e9kg, thanks for taking the time and effort to help us improve this work.
We understand that too much detail can make a reply harder to read, so we put a summary at the beginning of Part 1. You can skim it first. If anything is still a concern, you can find the details in our point-by-point replies below. We included these details because we hoped to address your concerns as fully as possible.
A brief summary
-
With the same inputs, Direct reached 11.66% mIoU in regions beyond the ego window, where Warp has no predictions. Combining the two improved full-window mIoU by about one point over Direct.
-
At the same spatial coverage, full QAMA outperformed the retrained simplified version by 5.10 points on S3 vehicle mIoU.
-
In 20 BEP-Pairs cases where only one neighboring vehicle’s plan changed, BEP-World captured some of the resulting changes in other vehicles’ occupancy (5.98% responder F1, versus 0% for rigid transport).
We’d be happy to hear your thoughts or discuss any remaining concerns.
[POST 3-1] Reply to Reviewer e9kg (Part 1/3)
We sincerely thank the reviewer for the constructive review. We are encouraged that the reviewer found that separating plan conditioning from spatial querying makes plan comparisons more flexible, addresses a limitation of ego-centric setups, and could be useful for cooperative driving and plan evaluation. In summary (more details in the point-by-point response):
Under identical inputs, joint conditional forecasting provides two things that a modular "forecast in the ego frame, then query" pipeline does not.
Modular comparison with matched inputs. We fix the weights, history, plans, and output resolution of the same multi-agent predictor and compare three inference paths. Table R3-1 shows the comparison of them. Warp is the modular alternative: forecast the future ego window, then resample it into the query. Direct forecasts the query window. Hybrid uses Warp inside the ego support and Direct elsewhere.
Table R3-1: S3, same seed-17 checkpoint, 4,117 origins. The metrics averaged over k = 1/3/7, ten non-ego queries, and 1/2/3 s.
As the reviewer notes (37.75 vs. 38.56 in the paper), Warp is slightly more accurate inside the ego window, and it is cheaper for many windows since it reuses one forecast. Our model provides this path as well, and Hybrid combines the two to gain about 1 pp Q-mIoU over Direct while keeping Direct's predictions outside the ego window. Direct is the fastest when a few regions are evaluated under many candidate plans, which is the use case that motivates the task.
Other modular references in the paper. CTG++ᴿ and CCDiffᴿ (Table 1c of the paper) are multi-agent trajectory predictors followed by shared occupancy rendering. They use additional inputs (maps and actor histories) and different training, so this comparison is system-level. BEP-World reaches 55.77 V-mIoU in S2 versus 45.58 / 45.59 for these two methods. We additionally render their predicted trajectories into the ten S3 non-ego query windows with the same shared renderer. They reach 43.65 / 43.63 V-mIoU with plan gains of +0.14 / +0.04 pp, compared with 49.77 V-mIoU and +1.12 pp for BEP-World (three-seed mean). Copy-Paste (persistence + plan transport + query readout), a training-free modular reference, reaches 46.64 V-mIoU (+0.12 pp) in S3. Despite their additional inputs, they trail BEP-World by 6.1–10.2 pp V-mIoU.
Coverage-matched modular alternative. Forecasting a larger ego-frame region that contains all queries and then cropping is Warp with a larger window, and it trades compute or resolution for coverage. Such a region needs about 94 m of extent, i.e., about 2.2× our XY tokens at 0.4 m on every request, or about 0.6 m cells at matched compute. At a comparable cost, Hybrid (1.90× Direct for one window) already combines Warp's full-resolution forecast inside the ego support with Direct's predictions outside it. A larger window also has coverage fixed at training time, whereas query conditioning spends a fixed budget at full resolution on whichever region is requested.