acceptodds
Withdrawn from ICLR 2027

Representation Selectivity in Interacting Systems

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning representations of interacting systems requires capturing not only the individual components, but also the relations between them. This becomes particularly important under restricted representation capacity, where not all information can be preserved, and the representation must select what is relevant. However, it remains unclear whether relational information should be formed before individual subsystem representations are compressed, or reconstructed afterward. In this work, we study how this choice interacts with the prediction objective, a phenomenon we call representation selectivity. We show that relational quantities can be represented compactly when information from both systems is available before compression, while separate compression may require preserving richer local information. Controlled nonlinear systems allow us to disentangle relational structure, physical coupling, and representation capacity. On the same interacting physical system, representations trained to predict future states show essentially no benefit from combining the two systems before compression, despite predicting the future state substantially better than a constant baseline. In contrast, when trained to predict force and relative motion at the physical interface, early access to both systems provides a large and consistent advantage. Our results show that, under restricted capacity, learning interaction-aware representations depends not only on architecture, but critically on what the representation is trained to predict.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, interacting systems, predictive representations, information compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VYAUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the difference between fusion before compression and separate compression followed by fusion under limited representation capacity, emphasizing that their relative advantages depend on the prediction target. The theoretical analysis provides a dimensional lower bound for recovering cross-system relations under separate compression and a target-relative information decomposition. Experiments on nonlinear oscillators, Lorenz systems, stochastic double-well interactions, and planar contact systems examine how system partition, prediction target, physical coupling, and representation width affect the performance of different fusion strategies.

Critical strengths
  1. The paper provides a clear theoretical argument with explicitly stated assumptions and limitations. It presents a useful perspective on how fusion strategies depend on the interplay among system partition, prediction target, and representation capacity.

  2. The paper conducts extensive controlled experiments across several dynamical systems, providing empirical evidence to support its main observations.

Critical weaknesses
  1. My main concern is that the contributions appear relatively incremental, while the empirical validation is largely limited to low-dimensional toy models. The distinction between early and late fusion has been extensively studied in multimodal learning, and the discussion of target-relevant information and representation capacity is closely related to established ideas in information bottleneck and distributed representation learning. For example, relevant works such as Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations (Mai et al., 2023) and Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework (Liang et al., 2023) are not discussed. Furthermore, the predominantly low-dimensional experimental settings make it difficult to assess how well the findings generalize to practical applications such as realistic embodied representation learning.

  2. There appears to be an inconsistency in Appendix Figure 4. At d=8d=8, the separate architecture still has an MSE close to the mean-prediction baseline, whereas the figure caption suggests that it approaches the performance of the joint architecture once each branch has sufficient width. This also seems inconsistent with the discussion in Appendix B.2.

Questions for authors
  1. How can the findings of this paper guide representation design in realistic embodied learning systems? Could the authors further clarify the practical implications beyond the controlled low-dimensional settings?

  2. Could the authors clarify the discrepancy between Appendix Figure 4 and its description? Why does the separate architecture still perform close to the mean-prediction baseline at d=8d=8, despite having sufficient representation width in principle?

Reviewer_axufOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper tackles the question of late vs early fusion in interacting systems and specifically asks how it is effected by the target objective and the representation capacity

Critical strengths
  • The paper explains well its scope and explains well the limitations
Critical weaknesses
  • The main research question is not really interesting. It is intuitively obvious that early fusion is preferred to late fusion if you need information that depends on the relation between two objects, unless you have enough capacity to preserve all the information. In the discussion they claim "Early versus late fusion is often treated primarily as an architectural choice. Our results show that its value cannot be assessed independently of the target and the representation budget." I think this is misstated. The architecture depends on the problem, which includes the target of what you are trying to learn.

  • The more general question of early vs late fusion has received a of work which isn't cited reducing the novelty significantly. For example “Early versus Late Fusion in Semantic Video Analysis”, "On the Benefits of Early Fusion in Multimodal Representation Learning", and "Attention Bottlenecks for Multimodal Fusion" to name a few.

  • The paper is very poorly written and really feels like AI writing, using a lot of nice sentences to not really say anything meaningful (e.g. 2.4 can be 1-2 sentences). Then there are parts that arent defined well in the paper (like G L_s and L_J not explained in Line 246), of the training process itself that is unclear. -The theory that doesnt really add any value, insight or novelty. Prop. 1 mostly says that to save information you need to be injective, I don't see the value it adds to the paper.

  • Small mistake - In eq 7 X⊥X_\perp is uncorrelated not independent

Questions for authors

NA

Reviewer_dD1AOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper seeks to investigate the early vs late fusion question in the presence of limited representational capacity, and how different setups with different losses and architectures affect the quality of learned representations.

Critical strengths

S1: Interesting problem and use of physical systems to set up controllable test environments.

Critical weaknesses

W1: Lack of novelty/poor grasp of existing literature in the space. Papers like [1] established that whether early vs late fusion would work better depends on the task, while the limited representational capacity described by the authors is the information bottleneck idea, investigated with regard to fusion stage by, e.g., [2], [3]. Furthermore, [4] and [5] investigate the synergy between input modalities relative to the output task. I think the authors will have to make major adjustments to the work in order to have sufficient novelty for publication.

W2: Main novel claim that isn't covered by existing literature is not adequately supported. Section 5 Y_end vs Y_port result is compelling, but multiple components (forecasting vs nowcasting, noise, number of dimensions, and dependence on partner history) are changing and these changes are not properly ablated.

[1] Snoek, Worring & Smeulders (2005), "Early versus Late Fusion in Semantic Video Analysis" [2] Barnum, Talukder & Yue (2020), “On the Benefits of Early Fusion in Multimodal Representation Learning” [3] Nagrani et al. (2021), “Attention Bottlenecks for Multimodal Fusion” [4] Liang et al. (2023), “Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework” [5] Hessel & Lee (2020), “Does my multimodal model learn cross-modal interactions?”

Questions for authors

Q1: Given existing literature in early vs. late fusion learning and the information bottleneck idea, can the authors justify the relative novelty of their contribution? Q2: Relatedly, these works are missing from the introduction/related work and should be added. Q3: Lower priority than the above, but in the reworking of the paper the authors should ablate the differences between Y_end and Y_port.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.