acceptodds
Under review as a conference paper at ICLR 2027

Bridge Transformer: Beyond Ego-Centered Fusion in Collaborative Semantic Occupancy Prediction

Abstract

Collaborative 3D semantic occupancy prediction enables multiple agents to exchange complementary information, yet, regardless of how many agents participate, the prediction remains a single semantic occupancy grid in the ego frame. The task is thus Many-to-One (M2O) in its output alone. Existing methods impose M2O on fusion as well. The resulting M2O fusion is centered on the ego representation and treats collaborator representations as inputs to be aggregated into it. Each collaborator representation, however, carries the errors of its single viewpoint. We empirically observe that M2O fusion passes these errors into the ego grid uncorrected, so the error rate rises in regions observed only by collaborators. This motivates Many-to-Many (M2M) fusion, which jointly refines the representations of all agents to correct these errors and defers the reduction to the ego grid until the output. To this end, we propose the Bridge Transformer, a feed-forward architecture for M2M fusion. Each agent represents its scene as 3D semantic Gaussians. These Gaussians are grouped, and each group is embedded into hierarchical tokens comprising an anchor token and its member tokens. Inter-agent attention over the anchor tokens of all agents alternates with intra-agent attention over each anchor and its members. Because every agent's anchor tokens serve as queries, this alternation jointly refines all agents' representations. A dual-branch Gaussian decoder then refines and extends each agent's Gaussians for splatting onto the ego grid. Experiments show that the Bridge Transformer outperforms the strongest prior method by 3.88 mIoU and 2.01 IoU. An ablation further shows that M2M fusion corrects more collaborator errors than M2O fusion.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
collaborative perception, 3D semantic occupancy prediction, 3D Gaussian splatting, transformer, autonomous driving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kypQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes Bridge Transformer, a many-to-many fusion framework that jointly refines hierarchical Gaussian representations across agents through alternating inter- and intra-agent attention before producing an ego-centric semantic occupancy prediction.

Critical strengths

The core idea of this paper is broadly applicable and potentially effective across many domains. The authors keenly identify this issue in the specific application field and develop a systematic method to address it, which is a notable strength of the work.

Critical weaknesses
  1. One limitation is that the evaluation is restricted to a single simulated benchmark, without validation on real-world data or under practical disturbances. This weakens the empirical evidence for the method’s generalizability and robustness.
  2. The reported results show substantial variation in performance gains across different classes, with the proposed method even performing considerably worse on some categories. I hope the authors could provide further analysis or explanation for these class-dependent differences.
Questions for authors

Please check the weaknesses and answer my questions.

4 comments
AuthorsOpenReview ↗

Response to Reviewer kypQ (1/2)

W1. Single simulated benchmark; no validation on real-world data or under practical disturbances.

Thank you for your comment on our evaluation. To our knowledge, no public real-world V2X dataset currently supports the evaluation of collaborative semantic occupancy prediction. Real-world V2X datasets, including DAIR-V2X [1], V2V4Real [2], and V2X-Real [3], are annotated only with 3D bounding boxes for vehicles, pedestrians, and other traffic participants. These boxes cannot be converted into dense semantic occupancy labels, because they do not cover static classes such as Road, Building, and Vegetation, and these datasets provide no point-wise semantic labels from which occupancy labels could be generated. This leaves Semantic-OPV2V as the standard benchmark for the task. CoHFF, VOGS-CP, and VQSOP all report results on this benchmark alone. Our method makes no simulation-specific assumptions, so it can be evaluated on real-world V2X data once such labels are available. We nevertheless agree with your point; our conclusion already acknowledges it and leaves validation on real data to future work.

Pose error is a key practical disturbance in collaborative perception because imperfect localization misaligns the features that agents share [4]. We therefore tested robustness to pose noise. Following the noise model of CoAlign [4], we added zero-mean Gaussian noise to each agent's position and heading, with σ ∈ {0.2, 0.4, 0.6} (m / °). The noise was added at inference only, and no model was retrained. CoHFF is not included because it does not spatially align collaborator features using the relative pose, so pose noise affects it through its learned pose embedding rather than through the feature misalignment that this protocol tests. Its authors report a GPS-noise study in their supplementary material.

Table R1. Robustness to pose noise at different noise levels σ, in mIoU (%, ↑). The Single-agent column reports results without collaboration, quoted from the VOGS-CP and VQSOP papers. Ours and Ours w/ mask are built on the VOGS-CP single-agent model and therefore share its value.

MethodSingle-agentσ = 0σ = 0.2σ = 0.4σ = 0.6
Ours29.2045.4242.4539.7738.06
Ours w/ mask29.2044.2441.2438.5136.92
VQSOP33.6141.5439.1136.4834.83
VOGS-CP29.2037.4436.7635.6934.88

Ours stays the most accurate at every noise level. It leads the stronger baseline at each level by at least 3.18 mIoU. It also leads Ours w/ mask by 1.14 to 1.26 mIoU across all noise levels, showing that the M2M gain remains stable under pose error. Ours shows a large decline as σ grows. Since pose noise affects only collaborator information, the decline suggests that Ours makes more effective use of this information and thus has more to lose. Even at σ = 0.6, the highest noise level, Ours still keeps a collaboration gain of 8.86 mIoU, above the clean-pose gains of 8.24 for VOGS-CP and 7.93 for VQSOP.

References

[1] H. Yu, Y. Luo, M. Shu, Y. Huo, Z. Yang, Y. Shi, Z. Guo, H. Li, X. Hu, J. Yuan, and Z. Nie. DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection. In CVPR, 2022.

[2] R. Xu, X. Xia, J. Li, H. Li, S. Zhang, Z. Tu, Z. Meng, H. Xiang, X. Dong, R. Song, H. Yu, B. Zhou, and J. Ma. V2V4Real: A Real-World Large-Scale Dataset for Vehicle-to-Vehicle Cooperative Perception. In CVPR, 2023.

[3] H. Xiang, Z. Zheng, X. Xia, R. Xu, L. Gao, Z. Zhou, X. Han, X. Ji, M. Li, Z. Meng, L. Jin, M. Lei, Z. Ma, Z. He, H. Ma, Y. Yuan, Y. Zhao, and J. Ma. V2X-Real: A Large-Scale Dataset for Vehicle-to-Everything Cooperative Perception. In ECCV, 2024.

[4] Y. Lu, Q. Li, B. Liu, M. Dianati, C. Feng, S. Chen, and Y. Wang. Robust Collaborative 3D Object Detection in Presence of Pose Errors. In ICRA, 2023.

AuthorsOpenReview ↗

Response to Reviewer kypQ (2/2)

W2. Performance gains vary across classes, and the method is considerably worse on some categories.

Thank you for your careful look at the per-class results. Our gains do vary across classes, but this is not unique to our method, since the gains of one baseline over another vary just as much. In Table 1, the per-class gain of VQSOP over VOGS-CP ranges from −11.32 to +21.70 IoU, and that of VOGS-CP over CoHFF from −30.55 to +38.60.

Despite this variation, the improvement is broad, as our method outperforms VQSOP, VOGS-CP, and CoHFF on 9, 10, and 8 of the 12 classes, respectively. Where it trails a baseline, the gap is at most 2.34 IoU. The only exception is CoHFF on three classes, Building, Pole, and Wall, where our method trails by 16.67 to 19.15 IoU. Even so, on Pole and Wall, our gap to CoHFF is smaller than those of VOGS-CP and VQSOP.

Table R2. Per-class IoU (%, ↑) on Building, Pole, and Wall. Values in parentheses denote each class's share of occupied voxels, measured by us on the validation set. Single-agent results are quoted from the CoHFF, VOGS-CP, and VQSOP papers, and collaborative results repeat Table 1. GSFormer is the single-agent model reported in the VOGS-CP paper. Ours and VOGS-CP share this model and differ only in fusion.

SettingMethodBuilding (5.49%)Pole (0.38%)Wall (2.04%)
Single-agentCoHFF21.0431.6633.30
GSFormer3.845.9430.79
VQSOP10.3612.2535.10
CollaborativeCoHFF25.7242.7458.27
VOGS-CP9.6112.1936.45
VQSOP11.2220.1636.51
Ours9.0523.5940.23

Table R2 shows that on Building and Pole, GSFormer is much weaker than single-agent CoHFF, trailing it by 17.20 and 25.72 IoU. Although our fusion narrows both gaps, this weaker single-agent starting point still leads to large final gaps. On Wall, in contrast, the two single-agent models differ by only 2.51 IoU, and the gap opens with collaboration: CoHFF gains 24.97 IoU, whereas Ours and VOGS-CP gain only 9.44 and 5.66. A likely explanation for all three gaps is that these classes are tall vertical structures, whose voxels are determined mainly by depth along each camera ray. Unlike CoHFF, which predicts per-pixel depth with a pretrained depth network, our method has no depth supervision, so the depth of its Gaussians along each ray is ambiguous. This ambiguity may also make the agents' observations of the same structure harder to align, which would limit the gain from collaboration.

Thanks for the response. The newly added experiments are useful for addressing my concerns. I have also checked the authors’ responses to the other reviewers. I decided to raise my score to 3.

Dear Reviewer kypQ,

Thank you for your response and for reading our replies. We are glad that the new experiments addressed your concerns. We sincerely appreciate your thoughtful review and your decision to raise the score.

Best regards,

Authors of Submission 40381

Reviewer_7kzkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Bridge Transformer for camera-based collaborative 3D semantic occupancy prediction. The core motivation is Many-to-One (M2O) is the task output but not necessary in the fusion stage, and baselines have substantially higher error in collaborator-only regions. They therefore propose Many-to-Many (M2M) fusion to jointly refine all agents' representations. Bridge Transformer groups semantic Gaussians into hierarchical anchor and member tokens, applies alternating inter-agent and intra-group attention, and uses a dual-branch decoder to refine existing Gaussians and generate additional ones. Experiments on Semantic-OPV2V report improvements of 3.88 mIoU and 2.01 IoU points over the strongest baseline, with lower communication efficiency.

Critical strengths
  1. Clear empirical motivation of substantial high error in collaborator region.
  2. Coherent architecture with hierarchical Gaussian tokenization to achieve affordable M2M cost.
Critical weaknesses
  1. The motivation might be overstated. An ego-centered aggregator can combine evidence from multiple collaborators and correct its output without explicitly updating their representations. Current methods fails to do so doesnot necessarily mean an inherent limitation of M2O.
  2. Table 3 indicates that Bridge Transformer is less efficient in CV as well as latency. It would be useful to further clarify accuracy-cost trade off by for example reducing number of Gaussians per agent from 25,600 to 12,800, 6,400. In addition, it would be useful to conduct ablation study for Gaussian-generation branch.
Questions for authors
  1. Could the authors clarify and explain the results with "Intra-only" and "Mask M2O"? It is a bit counter-intuitive that "Intra-only" with no information fusion has better performance than "Mask M2O".
2 comments
AuthorsOpenReview ↗

Response to Reviewer 7kzk (1/2)

W1. The motivation might be overstated: an ego-centered aggregator can correct its output without updating collaborator representations, so the failure of current methods does not necessarily imply an inherent limitation of M2O.

Thank you for your thoughtful comment on our motivation. Our Table 4 supports your observation that an ego-centered aggregator can correct its output. On the ≥1 correct voxels, where at least one collaborator's single-agent estimate is correct, M2O and M2M fusion perform comparably. However, the term "structural limitation" in our submission implies that every M2O design must have this limitation. Such a claim requires a theoretical proof, which our paper does not provide. Our experiments instead identify and quantify a consistent weakness across existing M2O methods. Therefore, to state our motivation more rigorously, we will replace "structural limitation" with "consistent weakness" throughout the final version.

The M2O models we evaluate have higher error rates in the collaborator region than in the ego region, 33.1% vs. 8.1% in Fig. 1a and 16.08% vs. 7.51% in Table 4. Most of this gap comes from the None correct voxels, where no collaborator's single-agent estimate is correct. There, the error rate reaches 59.5% and 28.63%. On the ≥1 correct voxels, it is only 8.4% and 4.45%, close to or below that of the ego region. This difference arises because M2O fusion refines only the ego representation with context from the other agents. That is enough when at least one collaborator is already correct. On the None correct voxels, however, each collaborator's error reaches the ego grid uncorrected, and the prediction is often wrong.

To test whether this weakness comes from the design of current methods rather than from M2O fusion itself, we compare Ours with Ours w/ mask, an ego-centered aggregator identical to Ours except for the attention mask. Removing the mask, which turns M2O fusion into M2M fusion, raises IoU by 1.10 and mIoU by 1.18 (Table 1). This gain comes from the collaborator region, where the error rate drops by 1.15 points overall and by 2.43 points on the None correct voxels; in the ego region, it changes by only 0.01 points (Table 4).

W2. Table 3 indicates that Bridge Transformer is less efficient in CV as well as latency. It would be useful to clarify the accuracy-cost trade-off, e.g., by reducing the number of Gaussians per agent from 25,600 to 12,800 and 6,400, and to ablate the Gaussian-generation branch.

Thank you for these concrete and constructive suggestions. We ran both experiments you suggested and report the results below. We will add them to the ablation study in the final version.

Table R1. Accuracy–efficiency trade-off for different numbers of Gaussians per agent (PP). CV and latency are defined as in Table 3.

PPCV (MB) ↓Latency (ms) ↓mIoU ↑IoU ↑
25,600 (default)2.46559.2 ± 252.045.4275.80
12,8001.23376.0 ± 172.444.1574.26
6,4000.61227.3 ± 98.344.0573.21

We retrained our model with PP = 12,800 and PP = 6,400. Halving PP from the default 25,600 to 12,800 halves the CV and reduces latency by 32.8%, which brings both close to those of VOGS-CP. The price is a drop of 1.27 mIoU relative to our default model. Halving PP again to 6,400 pushes the CV and latency below those of every baseline in Table 3 except VQSOP, yet costs only 0.10 more mIoU. At this setting, our latency is within 30.4 ms of VQSOP's, and our mIoU is still 2.51 higher. Over both steps, mIoU declines only gently, whereas IoU falls more likely because fewer Gaussians leave more of the occupied space uncovered.

Table R2. Ablation of the Gaussian-generation branch. JJ is the number of new Gaussians generated per anchor, and JJ = 0 removes the branch.

JJmIoU ↑IoU ↑
044.6375.06
145.2175.58
3 (default)45.4275.80
645.3475.87

Removing the Gaussian-generation branch lowers mIoU by 0.79 and IoU by 0.74, which confirms the effectiveness of this branch. With JJ = 1, mIoU already recovers 0.58 of these 0.79 points, suggesting that most of the gain comes from generating new Gaussians at all rather than from their number. Increasing JJ to 6 lowers mIoU by 0.08 and raises IoU by 0.07 relative to the default JJ = 3. A likely reason is that all JJ Gaussians decoded from one anchor token lie within the same bounded radius, so the extra Gaussians add no new information.

AuthorsOpenReview ↗

Response to Reviewer 7kzk (2/2)

Q1. Could the authors clarify and explain the results with "Intra-only" and "Mask M2O"? It is a bit counter-intuitive that "Intra-only" with no information fusion has better performance than "Mask M2O".

Thank you for your careful question about Intra-only and Mask M2O. The two variants differ from Ours only in the inter-agent attention block. Intra-only replaces it with another intra-agent attention block. Mask M2O adds an ego-centered mask to it, so that only the ego's anchor tokens attend to those of other agents. Therefore, strictly speaking, Intra-only still performs information fusion in the decoder: it decodes the Gaussians of all agents (the ego and the collaborators) with a shared decoder and splats them together onto the ego voxel grid to predict occupancy. The same holds for Mask M2O. To investigate why Intra-only outperforms Mask M2O, we compare the two variants in Table R3.

Table R3. Error rates (%, ↓) of Mask M2O and Intra-only in the regions of Fig. 1a. Mask M2O is Ours w/ mask, and its column repeats Table 4.

RegionMask M2OIntra-only
Ego region7.517.54
Collaborator region16.0815.91
  ≥1 correct4.454.26
  None correct28.6328.50

The masked inter-agent attention of Mask M2O gives the ego's anchor tokens cross-agent context. These anchor tokens are generated from the ego's own Gaussians, so they lie in the ego-visible region, where the error rate is already low (Fig. 1a and Table 4). The cross-agent context therefore has little left to correct, and the two variants have similar error rates in the ego region.

For the collaborators, however, the masked attention adds no new observation, because each collaborator's anchor tokens attend only to its own anchor tokens, which are all generated from the same agent's images. In contrast, Intra-only lets the anchor and member tokens of each collaborator refine one another twice as often as Mask M2O does. These tokens determine almost the entire output in the collaborator region, and this region has the highest error rate (Fig. 1a and Table 4). As a result, Mask M2O has a slightly higher error rate than Intra-only in this region, by 0.17 points, and a slightly lower mIoU.

Reviewer_oZfmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper points out that in collaborative semantic occupancy, the output is many-to-one (one ego grid), but existing methods also make the fusion many-to-one. Only the ego representation gets refined, and collaborator representations are used. So collaborator errors flow straight into the ego grid. The fix is many-to-many fusion, implemented as the Bridge Transformer.

Critical strengths

The problem framing is clear and easy to follow. The region-wise error-rate diagnostic in Fig. 1a is a nice, checkable way to motivate the work.

The results are strong. The method beats all prior methods, including the concurrent VQSOP, and the class-wise gains on thin structures (fence, guard rail, bridge) are real.

Critical weaknesses

The method has the largest communication volume (2.46 MB), higher latency than VQSOP.

M2M isn’t new to collaborative perception. V2VNet (ECCV 2020) already updates every agent’s hidden state through multi-round message passing.

Questions for authors

Why does M2M not help, in the “≥1 correct” region, which is exactly the case Fig. 1c is built around?

How does it compare with a V2X-ViT- or V2VNet-style M2M fusion applied to the same tokens?

4 comments
AuthorsOpenReview ↗

Response to Reviewer oZfm (1/2)

W1. The method has the largest communication volume (2.46 MB), higher latency than VQSOP.

Thank you for your helpful comment on efficiency. Our method does have the largest communication volume and a higher latency than VQSOP (Table 3). However, it achieves substantially higher accuracy, outperforming VQSOP by 3.88 mIoU (Table 1). The two methods have different goals. VQSOP is designed for communication efficiency and compresses each message with sparse-aware vector quantization. Our work instead studies how the received representations are fused, and each collaborator transmits all its Gaussians without compression. This is why our communication volume is the largest. Even so, it remains on the same order of magnitude as those of CoHFF (0.78 MB) and VOGS-CP (1.07 MB), and our latency is 42% lower than that of CoHFF.

Table R1. Accuracy–efficiency trade-off for different numbers of Gaussians per agent (PP). CV and latency are defined as in Table 3.

PPCV (MB) ↓Latency (ms) ↓mIoU ↑IoU ↑
25,600 (default)2.46559.2 ± 252.045.4275.80
12,8001.23376.0 ± 172.444.1574.26
6,4000.61227.3 ± 98.344.0573.21

The cost can also be reduced directly by using fewer Gaussians per agent. We retrained our model with PP = 12,800 and PP = 6,400. Reducing PP to 12,800 halves the CV to 1.23 MB, bringing it close to those of VOGS-CP (1.07 MB) and CoHFF (0.78 MB). Latency falls to 376.0 ms, already below that of VOGS-CP (378.7 ms). Yet mIoU drops by only 1.27 to 44.15, which is still 2.61 above that of the strongest baseline VQSOP, and the IoU of 74.26 also stays above that of VQSOP (73.79). Reducing PP further to 6,400 brings the CV down to 0.61 MB, below those of VOGS-CP and CoHFF. More importantly, the resulting latency of 227.3 ms is lower than those of all other baselines and within 30.4 ms of that of VQSOP (196.9 ms). mIoU drops by only another 0.10 to 44.05, still 2.51 above that of VQSOP, whereas IoU falls to 73.21, close to that of VQSOP. We will add this trade-off to the ablation study in the final version.

W2. M2M isn't new to collaborative perception. V2VNet (ECCV 2020) already updates every agent's hidden state through multi-round message passing.

Thank you for your helpful comment on related work. Your insightful point that M2M fusion is not new to collaborative perception is also a premise of our submission. The submission already cites both V2VNet and V2X-ViT in Sec. 1, and V2X-ViT again in Sec. 2. Accordingly, our contribution is to identify a consistent weakness of M2O fusion in existing methods for collaborative semantic occupancy prediction (CSOP) and therefore to introduce M2M fusion to this task. This step is far from trivial: the obstacle is computational cost.

The computational cost of any fusion grows with the size of one agent's representation times the number of agents refined. The task itself determines the first factor. 3D object detection, the task of V2VNet and V2X-ViT, outputs a sparse set of boxes, so a coarse BEV map (1.6 m cells with no height dimension) per agent suffices. CSOP instead outputs a label for every 0.4 m voxel of a 100×100×8 grid, 128 times as many cells as such a BEV map has over the same region, so each agent's representation must be far larger. The fusion paradigm determines the second factor, as M2O refines only the ego's representation, whereas M2M refines those of all agents.

Our hierarchical Gaussian tokenization and alternating attention make M2M fusion affordable for CSOP. Under the default setting, inter-agent attention runs over 512 anchor tokens per agent instead of 25,600 Gaussians, a 50-fold reduction in tokens and a 2,500-fold reduction in attention pairs. Intra-agent attention is confined to each anchor and its members, so its cost grows only linearly with the number of agents. Even with all agents refined, our latency is 42% lower than that of CoHFF. The cost can be reduced further with fewer Gaussians per agent. In Table R1 of our response to W1, using 6,400 Gaussians brings the CV down to 0.61 MB and the latency to 227.3 ms, both below those of VOGS-CP and CoHFF. Yet mIoU remains 2.51 above that of VQSOP, the strongest baseline.

AuthorsOpenReview ↗

Response to Reviewer oZfm (2/2)

Q1. Why does M2M not help in the "≥1 correct" region, which is exactly the case Fig. 1c is built around?

Thank you for your insightful question on the ≥1 correct voxels. To investigate why M2M fusion yields no gain on these voxels, we split them into All correct voxels, where every collaborator that estimates the voxel is correct, and Mixed voxels, where at least two collaborators estimate the voxel, their estimates disagree, and at least one is correct. The voxel in Fig. 1c is a Mixed voxel.

Table R2. Error rates (%, ↓) in the collaborator region by voxel type. The indented rows split the ≥1 correct voxels into All correct and Mixed, and the other two rows repeat Table 4.

Voxel typeOurs w/ mask (M2O)Ours (M2M)
≥1 correct4.454.58
  All correct4.724.93
  Mixed2.832.51
None correct28.6326.20

M2M fusion does help on the Mixed voxels, where it lowers the error rate by 0.32 points. This gain does not appear on the ≥1 correct voxels as a whole because the Mixed voxels make up only 14.46% of them and the error rate on the remaining All correct voxels rises slightly. Of the All correct voxels, 67.20% are observed by only one collaborator, which estimates them correctly. A likely reason for the rise is that the global attention in M2M fusion spans all agents' tokens, so the agents that do not observe such a voxel still update this collaborator's representation with their scene context, although its estimate of the voxel is already correct. These updates may occasionally alter a correct estimate, but they can also correct a wrong one. The error rate on the All correct voxels rises by only 0.21 points, whereas that on the None correct voxels falls by 2.43 points. This leaves a net reduction of 1.15 points in the collaborator region in Table 4.

The None correct voxels account for most of the errors of M2O fusion in the collaborator region and are therefore the focus of our paper. When designing Fig. 1, we chose a Mixed voxel as an intuitive example because we expected that a None correct case would be hard to follow at the very beginning of the paper. In such a case, one collaborator would predict car and the other truck, and the ego agent would start from a prior, yet the output would be the correct class, bus. However, to avoid the confusion you astutely point out, we will revise Fig. 1b and 1c to show a None correct voxel in the final version.

Q2. How does it compare with a V2X-ViT- or V2VNet-style M2M fusion applied to the same tokens?

Thank you for suggesting this comparison with V2VNet and V2X-ViT. We built two variants that replace only the inter-agent attention block of Ours and operate on the same anchor tokens. Cross-agent interaction in V2VNet and V2X-ViT is spatially local, so both variants need to define a neighborhood for each anchor token. We divide the ego frame into 1.6 m × 1.6 m × 3.2 m pillars, and the neighborhood is the 3×3 block of pillars centered on the one that contains the anchor token's position. Both settings are taken from the original configurations of the two methods. The V2VNet-style variant runs V2VNet's message passing between each anchor token and the other agents' anchor tokens in its neighborhood, with an MLP and a GRU replacing the message CNN and the ConvGRU. In the V2X-ViT-style variant, each anchor token attends first to all anchor tokens in its neighborhood and then to all anchor tokens of its own agent, followed by an FFN. These two attention steps stand in for HMSA and MSwin, respectively.

Table R3. Comparison of M2M fusion designs.

VariantmIoU ↑IoU ↑
Ours w/ mask44.2474.70
V2VNet-style44.5774.81
V2X-ViT-style45.0175.28
Ours45.4275.80

Both variants surpass Ours w/ mask by 0.33 and 0.77 mIoU, which is consistent with the advantage of M2M fusion over M2O fusion. Ours outperforms the V2VNet-style and V2X-ViT-style variants by 0.85 and 0.41 mIoU and by 0.99 and 0.52 IoU. A likely reason is the interaction range. In both variants, each cross-agent interaction limits an anchor token's context from other agents to its neighborhood, a 4.8 m × 4.8 m window. In contrast, our inter-agent attention draws this context from all anchor tokens across the entire scene.

Thank you for addressing my concerns, I keep my positive score.

Dear Reviewer oZfm,

Thank you for reading our responses so carefully and for your continued support of our paper. Your comments have made the paper clearer and more complete. We truly appreciate the time and care you have given to our work.

Best regards,

The Authors

Comments on the paper

3 comments

Dear Reviewers,

The rebuttal deadline is approaching. I am leaving this friendly reminder to encourage active discussion. Thank you for your valuable time and engagement!

AC

AuthorsOpenReview ↗

Summary of Revisions for All Reviewers

We thank the area chair and the reviewers for their valuable comments. We have now uploaded a revised PDF, in which new and rewritten text is shown in blue. This revision already includes the changes that our responses promised for the final version:

  1. Wording of the introduction (Sec. 1, pp. 2–3). Paragraph 3 and Contribution 1 now describe M2O fusion in existing methods as having "a consistent weakness." (Reviewer 7kzk, W1)
  2. Fig. 1(b, c) and its caption (p. 2). The example is now a None correct voxel. Both collaborators misclassify this bus voxel, one as car and the other as truck, and M2M fusion corrects both to bus. (Reviewer oZfm, Q1)
  3. Accuracy–cost trade-off (Table 3 and Sec. 5.1, p. 8). We added mIoU and IoU columns to Table 3, as well as two rows for our model with PP = 12,800 and 6,400 Gaussians per agent. A new paragraph, the second under "Efficiency," analyzes these results. We placed them next to the efficiency comparison rather than in the ablation study. (Reviewer 7kzk, W2; Reviewer oZfm, W1)
  4. Ablation of the Gaussian-generation branch (Table 5 and Sec. 5.2, p. 9). The paper calls this branch the anchor branch. We added a JJ column and rows for JJ = 0, 1, and 6 to Table 5. A new paragraph, "Generating new Gaussians from anchor tokens," analyzes these results. (Reviewer 7kzk, W2)
  5. Robustness to pose noise (Appendix B, p. 14). We added Table 6, which reports mIoU under pose noise with σ\sigma = 0.2, 0.4, and 0.6, together with an analysis. The noise model follows CoAlign (Lu et al., 2023), which we added to the references. Sec. 5.1 points to this appendix. (Reviewer kypQ, W1)

Again, we sincerely thank all reviewers, ACs, and PCs for their time and effort on our submission. We welcome any questions during the discussion period.

AuthorsOpenReview ↗

Summary of the Author-Reviewer Discussion

We sincerely thank the reviewers and the area chair for their time and constructive feedback. The reviews recognize the problem identified (Reviewer kypQ), its empirical motivation (Reviewers 7kzk and oZfm), the architecture that makes M2M fusion affordable (Reviewer 7kzk), the strong results over all prior methods (Reviewer oZfm), and the broad applicability and potential of the core idea across many domains (Reviewer kypQ).

We summarize the main points and our responses below:

  • Claim of an inherent or structural limitation of M2O fusion (Reviewer 7kzk): We agree that the weakness of M2O fusion should not be called a "structural limitation," since that would require a theoretical proof. The revised paper calls it a consistent weakness across existing M2O methods, which is what our experiments identify and quantify. Table 4 compares Ours with its M2O variant, Ours w/ mask, and this comparison supports linking the weakness to M2O fusion rather than to other design choices.
  • ≥1 correct voxels and the example in Fig. 1(b, c) (Reviewer oZfm): In our response, we split the ≥1 correct voxels into All correct and Mixed voxels, compared their error rates, and explained in detail the likely reason why M2M fusion shows no overall gain on them. Because Table 4 shows that the gain of M2M fusion is concentrated on the None correct voxels, the revised paper replaces the ≥1 correct voxel shown in Fig. 1(b, c) with a None correct voxel and updates its caption.
  • Novelty of M2M fusion (Reviewer oZfm): We agree that M2M fusion is not new to collaborative perception. Our contribution, as the submission states, is to bring it to collaborative semantic occupancy prediction, and our response explains why the computational cost of applying M2M fusion to this task is the obstacle. The submission also cites V2VNet and V2X-ViT (Secs. 1 and 2), and during the rebuttal we further built a variant in the style of each. Both variants surpass Ours w/ mask, consistent with the advantage of M2M over M2O fusion. Ours outperforms both, likely because its inter-agent attention spans all anchor tokens in the scene.
  • Intra-only versus Mask M2O (Reviewer 7kzk): Our response explains how the two variants differ and why, strictly speaking, Intra-only still performs information fusion. It also adds a comparison of their error rates by region, which offers a plausible explanation for why Mask M2O (Ours w/ mask) has a slightly lower mIoU than Intra-only.
  • Ablation of the Gaussian-generation branch (Reviewer 7kzk): The revised paper adds an ablation of this branch and shows that it improves both mIoU and IoU. It also evaluates how this gain varies with the number of generated Gaussians.
  • Communication volume and latency (Reviewers 7kzk and oZfm): Table 3 of the revised paper now also reports the accuracy–efficiency trade-off for different numbers of Gaussians per agent. Using fewer Gaussians lowers both costs. With 6,400 Gaussians per agent, for example, both costs fall below those of CoHFF and VOGS-CP, while mIoU remains well above those of all three baselines.
  • Single simulated benchmark and practical disturbances (Reviewer kypQ): Our response explains why no public real-world V2X dataset supports this task, and CoHFF, VOGS-CP, and VQSOP are likewise evaluated only on Semantic-OPV2V. Validation on real data is therefore left to future work, as the conclusion states. We also add an evaluation under pose noise in Appendix B, with Gaussian noise added to each agent's pose at inference only. At every noise level, our method remains well above Ours w/ mask, VOGS-CP, and VQSOP.
  • Variation of gains across classes (Reviewer kypQ): Per-class gains vary as much between the baselines as for our method, which outperforms each baseline on most classes (Table 1). Our method trails CoHFF by a wide margin on Building, Pole, and Wall. Our analysis of these three classes during the rebuttal suggests that the gaps come from a weaker single-agent starting point on Building and Pole, and from the lack of depth supervision, which matters for such tall vertical structures.

We again sincerely thank the reviewers, the area chair, and the program chairs for their time, effort, and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.