acceptodds
Under review as a conference paper at ICLR 2027

GroundingV2X: Grounded Pre-Training for Model-, Task-, and Alignment-Agnostic Collaborative Perception

Abstract

Collaborative perception aims to enhance the perception capability of a single vehicle by enabling information sharing through Vehicle-to-Everything (V2X) communication. However, the heterogeneity of agents poses a significant challenge to the scalability of collaborative perception systems. Existing approaches rely on encoder retraining, feature interpretation, or protocol adaptation to align heterogeneous representations, which are difficult to implement in real-world practice. To address this, we introduce GroundingV2X, a model-, task-, and alignment-agnostic collaborative perception framework via grounded pre-training. Specifically, agents communicate through grounded text derived from their perception results and composed of positional, geometric, and semantic key-value pairs with numerical attributes, providing a unified interface for heterogeneous information exchange. Moreover, we incorporate Fourier numerical embeddings and sub-sentence level attention to preserve numerical precision and entity-specific context with a lightweight text encoder. Then, a position-aware text-to-feature enhancer selectively integrates the grounded text into spatially relevant local features for collaborative perception. In addition, each agent is pre-trained independently using its perceived grounded text, synthetic grounded text, and annotation-derived pseudo-labels for supervision, improving its generalization to diverse textual descriptions. Extensive experiments on OPV2V, DAIR-V2X, and V2V4Real demonstrate that GroundingV2X enables efficient heterogeneous collaboration solely through pre-training while achieving superior performance compared to state-of-the-art methods both in single-task and multi-task scenarios with substantially lower communication bandwidth requirements.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Collaboration Perception, Heterogeneous Collaboration, Grounded Pre-Training

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_YtXpOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposed grounded language as a new communication interface for heterogeneous collaborative perception. As the feature space of each kind of agent is directly mapped to a formatted language, no co-adaptation is needed. Results on multiple datasets show competitive perception performance on top of the paradigm's benefits.

Critical strengths

The proposed method is intended to address the challenge of "how to make a model available for collaboration without knowing the collaborator". This primary goal is achieved by mapping universal structured language to an individual feature space.

Critical weaknesses
  1. The claim of voiding some groups of methods from the comparison is not well-supported. While the language is naturally a universal interface, other learning-based interfaces are not naturally non-universal.

Feature-based methods like GT-Space (ICLR26)/GenComm (NIPS25) and codebook-based methods like CodeFilling (CVPR24)/CodeAlign (CVPR26) can be adapted to the setup by fixing a feature space/code trained from a single configuration and translating later configurations into this fixed feature space/code.

  1. Due to the significant latency overhead introduced by the relatively complicated feature2text + text2feature conversion, the benchmarked performance and latency robustness are not a fair comparison.
Questions for authors
  1. What is the motivation for choosing structured language over parameterized representation?
  2. Is the text2feature and feature2text time symmetric? Is the BEV agent's overhead usually higher or lower?
Reviewer_DU8gOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces GroundingV2X, a heterogeneous collaborative perception framework that lets agents with different sensors, model backbones, and perception tasks work together without cross-agent feature alignment or joint post-training. Instead of sharing intermediate features, it uses grounded text with structured semantic, geometric and positional attributes as a unified communication interface. To properly fuse incoming messages, the authors design Fourier numerical embeddings, sub-sentence attention, and a position-aware text-to-feature enhancement module. They further propose a grounded pre-training scheme built on synthetic cross-task descriptions and annotation-derived pseudo-labels. This allows each agent to learn the communication interface separately, so independently trained agents can cooperate without extra joint tuning. Evaluations on OPV2V, V2V4Real and DAIR-V2X show solid results in heterogeneous settings, especially when agents run different perception tasks. The method also cuts communication overhead significantly compared with intermediate feature fusion.

Critical strengths
  1. The problem is practically relevant. Collaborative perception systems often need to integrate heterogeneous agents with different sensors, models and task objectives. Prior methods usually require feature alignment or custom adaptation, which makes it hard to combine separately developed agents. This work tries to remove such constraints and offers a potentially scalable path for heterogeneous collaboration.
  2. Independent pre-training without joint post-training is a nice property. Each agent learns to parse and merge grounded descriptions on its own, with no access to partners’ internal features or weights. Once trained separately, heterogeneous agents can be deployed together without further cross-agent optimization. This fits well for real systems where agents may be updated independently.
  3. It works across different perception tasks. The experiments show the framework supports information sharing between 3D detection and BEV segmentation agents. In Table 3, GroundingV2X consistently outperforms single-agent baselines and STAMP across the tested heterogeneous setups. The results suggest grounded descriptions can bridge different perception tasks without a shared feature space.
  4. The pre-training design is well supported by ablation studies. Synthetic text and pseudo-labels expose each agent to diverse description formats during pre-training. Table 6 shows removing this synthetic pre-training hurts cross-task collaboration a lot, while same-task performance stays stable. This clearly highlights the importance of the pre-training strategy for cross-task compatibility. Additional tests on numerical encoding, incomplete descriptions, pose noise and communication latency add further support to the analysis.
Critical weaknesses
  1. It remains unclear why grounded text is preferable over standard structured representations. The messages exchanged are essentially predefined key-value entries encoding semantics, positions and geometry — similar to structured object-level data widely used in collaborative perception. While the method beats simple late fusion, these comparisons do not prove that language-style encoding is required for alignment-free collaboration. One could use an MLP or Transformer to encode structured object/region records and feed them into the same position-aware fusion pipeline while keeping the independent pre-training setup. A controlled comparison with identical attributes, supervision and fusion logic would help isolate the unique value of grounded text itself.

  2. The claimed task-agnostic generalization is limited to the two tested tasks. The paper only evaluates 3D detection and BEV segmentation, and the synthetic descriptions used in pre-training are tailored to these two tasks. The results therefore only confirm compatibility among independently trained agents for known task types, not generalization to unseen tasks or new message schemas. The large drop in cross-task performance after removing synthetic pre-training also implies that compatibility relies heavily on prior exposure to the corresponding description format. The authors should clarify the actual scope of “task-agnostic” collaboration. If feasible, testing on an unseen task without retraining the receiver would strengthen the claim.

  3. Real-world performance is not competitive enough. GroundingV2X does well on OPV2V but falls behind on real-world datasets. On DAIR-V2X, it reaches [email protected] = 0.655, whereas HEAL gets 0.770 and STAMP gets 0.758. It is also slightly worse than HEAL on V2V4Real. The authors attribute the DAIR-V2X gap to inconsistent annotations across agents, but this claim lacks controlled evidence. Since heterogeneous real-world deployment is a central motivation, it would be useful to investigate whether the grounded representation discards information that feature-level fusion can preserve.

  4. Individual module contributions are not fully disentangled. The paper ablates numerical encoding and synthetic pre-training, but leaves several key components unstudied: sub-sentence attention, position-aware routing, and different grounding objectives. Moreover, the synthetic pre-training ablation removes synthetic descriptions and pseudo-labels together, so we cannot tell their separate impacts. More fine-grained ablations would help identify which components drive the observed gains.

  5. Communication and deployment costs need clearer reporting. Stating communication cost is roughly zero MB per agent is not informative; exact payload sizes would be much more useful. The text encoder is BERT-base, and latency is measured on an H100, which is not representative of resource-limited automotive hardware. Reporting exact payloads, bandwidth-matched comparisons, and end-to-end latency under tighter compute constraints would make the deployment claims more convincing.

Questions for authors

My main concerns are the insufficient justification for grounded text against simpler structured representations, the narrow scope of task-agnostic evaluation, and the performance gaps on real datasets. It is unclear why grounded text is necessary and show stronger generalization beyond the current task setup. Please refer to weaknesses.

Reviewer_tTkdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces GroundingV2X. Each agent, either a connected vehicle or a roadside unit, converts its detection results or bird’s-eye-view (BEV) segmentation outputs into structured text—data composed of key-value pairs, where numerical values ​​are encoded using Fourier embeddings. The receiving vehicle uses BERT to encode this text and integrates it into its own BEV features via a similarity-based routing module. During training, the perception encoder of each vehicle remains frozen. The training data consists of the vehicles' own outputs from a cooperative perspective, as well as synthetic statements constructed based on dataset annotations. The research topic is practically significant, however, I do not believe this text-based representation is necessary. Furthermore, the experimental results in practical settings are not as favorable as those claimed in the abstract, and the paper fails to demonstrate the system's robustness against cross-modal errors encountered in real-world scenarios.

Critical strengths

Detection and segmentation agents can collaborate without exchanging dense features and without joint post-training. This is attractive in practice. Experiments cover both simulated and real-world data. The ablations on numerical encoding, synthetic cross-task text, and dropped or permuted key-value pairs are informative.

Critical weaknesses
  1. The messages are box or rectangle attributes that follow a fixed grammar. Every number is replaced by [NUM] plus FoNE, so the language part is little more than key names and a category word. Table 7 compares numerical encodings, but all of them run inside BERT. There is no baseline without a language model, such as a per-box MLP or set encoder that uses the same routing, enhancer, training data and losses. The bandwidth saving comes from sending perception results instead of features, not from using text. Late fusion is also ≃0, and a binary packet would be smaller than a text string. Actual message sizes in bytes should be reported.

  2. Camera and LiDAR detections may assign substantially different positions to the same object. The proposed routing matches message geometry to spatial cells, but does not explicitly resolve object correspondence or localization uncertainty. An incorrect yet well-formed message could therefore enhance features at the wrong location. Pose-noise experiments do not adequately isolate this issue.

  3. The real-world experiments use LiDAR–LiDAR configurations, leaving real-world camera–LiDAR collaboration unverified. The implementation also relies on spatial grids and detection/segmentation supervision; applicability to other representations or end-to-end driving models remains untested.

Questions for authors

How does the method compare with a structured box/region encoder that uses the same fusion architecture, training data and supervision?

When two agents report non-overlapping boxes for the same object, what prevents duplicate detections or feature injection at the wrong location? Can this be evaluated with controlled object-level localization errors?

Can you provide real-world camera–LiDAR results? What does a new agent need in order to join without retraining the existing agents (shared grammar, annotations for other tasks, a BEV grid)?

With up to 40 sentences per collaborator, the concatenated sequence can exceed BERT’s 512 positions. How is this handled? Since cross-sentence attention is masked, how does this differ from encoding each sentence separately?

Can you give the STAMP reproduction details behind Table 3? More generally, please state for each table whether baseline numbers are taken from the original papers or retrained by the authors, and under which settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.