Representation Selectivity in Interacting Systems

Learning representations of interacting systems requires capturing not only the individual components, but also the relations between them. This becomes particularly important under restricted representation capacity, where not all information can be preserved, and the representation must select what is relevant. However, it remains unclear whether relational information should be formed before individual subsystem representations are compressed, or reconstructed afterward. In this work, we study how this choice interacts with the prediction objective, a phenomenon we call representation selectivity. We show that relational quantities can be represented compactly when information from both systems is available before compression, while separate compression may require preserving richer local information. Controlled nonlinear systems allow us to disentangle relational structure, physical coupling, and representation capacity. On the same interacting physical system, representations trained to predict future states show essentially no benefit from combining the two systems before compression, despite predicting the future state substantially better than a constant baseline. In contrast, when trained to predict force and relative motion at the physical interface, early access to both systems provides a large and consistent advantage. Our results show that, under restricted capacity, learning interaction-aware representations depends not only on architecture, but critically on what the representation is trained to predict.