acceptodds
Withdrawn from ICLR 2027

Communication-Efficient Multi-Task Collaborative Perception Using Joint Representation Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Collaborative perception (CP) improves the environmental understanding of connected agents via the exchange of visual scene representations. While prior works focus on single-task perception, autonomous agents often need to run multiple tasks concurrently, for comprehensive scene understanding. A direct application of existing CP frameworks requires each agent to send a separate feature representation for every task, causing communication to scale poorly with both the number of tasks and agents. We study multi-task CP from a joint representation-sharing perspective. Intuitively, task-specific features produced by an agent are distinct encodings of the same local observation and hence, contain overlapping scene information. We formalize this using rate-distortion theory, comparing separate task-wise coding with joint coding of multiple task-specific feature representations. The analysis shows that, for the same task-level distortion requirements, jointly coding multiple task representations can reduce communication by leveraging the information shared among them. Motivated by this insight, we develop ORAT (One Representation for All Tasks), a multi-task CP framework that maps multiple task-specific representations into a single base representation before transmission. Unlike prior task-wise communication methods that apply compression, task-relevant region selection, or receiver-driven refinement on each task-specific representation separately, ORAT applies these operations directly on that base. The receiver agent adapts the shared base representation into task-compatible features through learned task-specific projection modules, and fuses them with its local features for downstream prediction. Experiments on OPV2V and V2X-Sim datasets show that ORAT preserves perception performance for all tasks while reducing communication by factors ranging from 3.61× to 4810× compared to prior task-specific transmission frameworks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Cooperative perception, communication-efficient learning, information theory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gF2pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates communication redundancy in multi-task collaborative perception, uses rate-distortion analysis to show that jointly encoding multiple task representations can reduce communication overhead, and proposes ORAT, a framework that aggregates task-specific features into a single shared representation. By combining quantization, region selection, and receiver-driven residual refinement, ORAT achieves comparable multi-task perception performance on OPV2V and V2X-Sim while substantially reducing communication volume.

Critical strengths
  1. The problem is practically relevant. The paper addresses redundant communication in multi-task collaborative perception, better reflecting the need for autonomous systems to perform multiple perception tasks simultaneously.

  2. The approach has theoretical support. Rate-distortion analysis explains the potential communication gains from joint encoding, while conditional total correlation characterizes shared information across tasks.

  3. The framework is well integrated. It combines shared representations, layered quantization, task-relevant region selection, and receiver-driven residual refinement to address information redundancy both across tasks and across agents.

Critical weaknesses

Overall, the multi-task representation problem studied in this paper is important for collaborative perception. However, the proposed method and its experimental validation are not yet sufficiently convincing.

  1. The method uses an encoder–decoder architecture to generate a unified representation, an approach commonly adopted in heterogeneous collaborative perception. The authors should further clarify its advantages and validate the contribution of joint multi-task encoding itself. Moreover, the current architecture relies on a predefined set of tasks, and its extensibility remains unclear: does adding a new task or missing the input features for an existing task require retraining?

  2. The experimental evaluation is not sufficiently comprehensive. For detection performance, I recommend reporting the stricter AP@70 in addition to the relatively lenient AP@50. For baselines, representative unified representation methods from other fields should be included. Runtime results for the compared methods should also be reported under the same hardware conditions. Furthermore, ORAT involves multiple rounds of communication, so an end-to-end latency comparison accounting for computation, feedback, and transmission overhead is needed. The delay robustness experiment in the appendix does not replace this evaluation.

  3. The advantages of the method remain unclear. ORAT does not consistently outperform the baselines in perception performance, and some tasks exhibit performance degradation. Figure 3 has poor readability and reports only ratios; separate communication volumes and performance–communication curves should be provided for the main methods. The tables also bold ORAT’s results even when they are not the best, which may cause confusion. The highlighting convention should be clarified.

  4. Communication accounting and the attribution of gains should be made clearer. Quantization and entropy coding are not exclusive to unified representations and can also be applied to task-specific representations. Although some baselines already use these techniques, joint and separate encoding should still be compared under compression settings that are as consistent as possible to isolate the gains from the unified representation itself. The authors should also clarify whether communication volume includes overhead such as region coordinates or masks and residual requests, and how broadcast and per-receiver transmissions are counted, to avoid potentially unfair comparisons.

Questions for authors

None

Reviewer_zBiZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Firstly, this paper introduce a framework that using multi-task CP from a joint representation-sharing. Secondly, all efficient communication operations, including quantization, task-related filtering, and receiver condition refinement, are carried out on joint representation, rather than on individual task-specific streams. Finally, the receiver adjusts the refined shared foundation to task-compatible features through a lightweight task-specific projection module, and these features are used for downstream task-specific inference.

Critical strengths
  1. The paper addresses a practical issue of communication, transmitting overlapping information separately for detection and segmentation. The author propose a compact joint-representation message across tasks.
Critical weaknesses
  1. Firstly, the ablations in Fig. 6 only cover Huffman coding and the residual threshold. From my perspective, the author need to do more ablation experiments, such as, providing the ablation tables, evaluating the model with and without each proposed modules. Without that, I can’t really figure out the real gain from the proposed modules.
  2. Secondly, the communication/accuracy trade-off needs a more direct comparison. For example, in Table 2, the proposed ORAT gets dynamic IoU 49, but the baseline CoBEVT is 54. The lower communication may justify that drop, but if the author claims performance is preserved, it should state an explicit tolerance for each task. Moreover, I also notice the Fig. 3 metric, the performance is computed by log₂(Mbits), however, changing the communication unit may change the ranking? Hence, the author should better provide a AP/IoU against varying bits per frame? It would be much easier to understand. Also, please clarify whether the requests and residual messages are included in the communication?
  3. The collaborative agent number experiments are missing. I am very curious to see the figure of the proposed ORAT and other baselines’ AP/IoU performance under different CP agent numbers.
  4. Finally, the ORAT are evaluted through the simluated OPV2V and V2X-Sim datsaets. The delay experiment is useful, but it doesn’t show whether the proposed joint representation works under the real world setting, such as, unpredicted noise and localization errors. So, at least one real-dataset (V2V4Real, etc) is needed to support the proposed claims. Additionly, please report Table 2 with one or two decimal places.
Questions for authors

Please see the weakness.

Reviewer_TRpsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes to use a shared representation for multiple perception tasks (when all the tasks are executed). Multiple components are included to enhance the communication-perception trade-offs.

Critical strengths
  1. The contribution is clearly scoped to one representation for multiple perception tasks. The problem setting is valuable as the variety of tasks in autonomous driving/robotics rapidly increases.
  2. The quantitative results show that ORAT achieves the best communication-perception trade-offs in its own setting.
Critical weaknesses

While the problem setting and motivation are interesting and valuable, the empirical evidence is far from sufficient to provide enough insight into the problem.

  1. Despite Section 4 providing some proof and theorems to support the design, the only major module that enables the ORAT is the aggregator, which aggregates multiple representations into one smaller representation. All the later modules are irrelevant to the "joint representation" part of the contribution. However, there is no detailed discussion of how the aggregator is designed, and how different designs can lead to different results. Quantization, masking, and entropy coding have been used in collaborative perception so that I do not consider them as major technical contributions (the ablation study is done on these modules).

  2. The experimental results are not presented rigorously. ORAT is bolded in Tables 1 and 2, but it is not the best method on any single metric. This could be misleading, though the author did not say "bold means best". There is also a numerical conflict between Table 2 and Table 3 (36 vs 35 on static IoU).

Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.