Position: Interaction Understanding Requires More Than Object-Centric Representations

Interaction understanding should be an explicit target of representation learning for agents that act in the physical world. A system can maintain object identities and recognize an interaction category while leaving open how a particular relationship is developing: whether a grasp is being established or released, whether a handover has begun, when the current contact will end. We argue that the state of the interaction between particular participants, and how it changes, should be an explicit learning target rather than an assumed byproduct of object perception or interaction recognition, and that progress should be judged by whether that state transfers to new questions about the event. A pilot study with SAM4D, a small readout trained on frozen SAM 3 features and told which two participants to attend to, illustrates the distinction. Telling the readout both participants raises relation Recall@3 by 19.39 percentage points over an actor-only query from the same features. On 15 events from eight videos, however, the recognition-trained representation shows no reliable advantage over controls with the same head, nonvisual inputs, and training budget at predicting how much time is left in the current event, even though the same classifier, replayed on the same frames, still ranks the recorded label in its top three on 44.80% of them, averaged over videos. Two human annotators who did not see the automatic labels both record 9 of the 15 events, with start and end times that differ from the automatic ones by a median of 0.2 s; on these nine events the label-only duration prior still has the lowest error. In this pilot, recognizing an interaction and knowing its current state come apart. Whether objectives that target this state transfer better is untested; we specify the matched comparisons of learning objectives that would decide it, and what each outcome would mean for the position.