acceptodds
Under review as a conference paper at ICLR 2027

Position: Interaction Understanding Requires More Than Object-Centric Representations

Abstract

Interaction understanding should be an explicit target of representation learning for agents that act in the physical world. A system can maintain object identities and recognize an interaction category while leaving open how a particular relationship is developing: whether a grasp is being established or released, whether a handover has begun, when the current contact will end. We argue that the state of the interaction between particular participants, and how it changes, should be an explicit learning target rather than an assumed byproduct of object perception or interaction recognition, and that progress should be judged by whether that state transfers to new questions about the event. A pilot study with SAM4D, a small readout trained on frozen SAM 3 features and told which two participants to attend to, illustrates the distinction. Telling the readout both participants raises relation Recall@3 by 19.39 percentage points over an actor-only query from the same features. On 15 events from eight videos, however, the recognition-trained representation shows no reliable advantage over controls with the same head, nonvisual inputs, and training budget at predicting how much time is left in the current event, even though the same classifier, replayed on the same frames, still ranks the recorded label in its top three on 44.80% of them, averaged over videos. Two human annotators who did not see the automatic labels both record 9 of the 15 events, with start and end times that differ from the automatic ones by a median of 0.2 s; on these nine events the label-only duration prior still has the lowest error. In this pilot, recognizing an interaction and knowing its current state come apart. Whether objectives that target this state transfer better is untested; we specify the matched comparisons of learning objectives that would decide it, and what each outcome would mean for the position.

Primary area:
applications to neuroscience & cognitive science
Keywords:
interaction understanding, representation learning, object-centric representations, human-object interaction, video understanding

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_o3NJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper argues that it is difficult to adequately understand interaction states that change over time using only existing object-centric representations and interaction recognition, and that these should be treated as explicit learning targets. To support this, we present experimental results showing that participant conditioning improves relation recognition performance using SAM4D based on SAM 3, but the trained representations do not demonstrate a consistent advantage over the baseline in predicting the remaining-time of interactions.

Critical strengths
  1. This paper raises a meaningful problem by distinguishing interaction recognition from interaction-state understanding and emphasizing the importance of event-level reasoning and downstream transfer.
  2. This paper strive to ensure the reliability of experimental results through various controls and analyses, such as random readout, non-visual baselines, and human annotation audits.
Critical weaknesses
  1. Unverified Central hypothesis : The main argument is that explicit interaction-state learning provides better transfer, but the study did not actually learn the objective or compare it with existing objectives. Therefore, there is a lack of direct experimental evidence supporting the paper's main argument.
  2. Insufficient experiment : The temporal evaluation consisted of only 8 videos and 15 events, and the major performance differences were not statistically significant. Therefore, it is difficult to conclude that recognition and interaction-state understanding are actually separate.
  3. Confusion in temporal evidence : It is difficult to distinguish whether the failure of remaining-time prediction is due to the limitations of the representation or the limited causal head and temporal observations. In particular, event termination can rely on an unobservable human intention, which limits its use as an evaluation metric for understanding interactions.
Questions for authors

see weakness

6 comments
AuthorsOpenReview ↗

Response to Reviewer o3NJ (Part 1/3)

Dear Reviewer o3NJ,

Thanks for your valuable comments! We are glad that you found the problem meaningful. Our main message is twofold:

  1. We followed your comments. We train the interaction-state objective and compare it with three existing objectives under matched conditions (W1), and we add a question that does not depend on an unobservable intention (W3).
  2. The result supports the first part of our position. At pilot scale, replacing relation-recognition training with explicit interaction-state supervision did not improve transfer to a different event-level question. This suggests that our original negative result is not simply an artifact of training only on relation recognition. More broadly, it supports testing transfer directly: the ability to answer event-level questions should not be assumed from the training objective or from success on the training task. It has to be demonstrated on held-out event-level questions.

We address the weaknesses below. Please kindly let us know if you have further concerns.

W1: Unverified Central hypothesis

Response 1: Thank you for the valuable feedback on our central hypothesis, and we apologize for any confusion about the main argument. We here clarify that the position has two parts (Section 1, lines 087–091). The first says what to aim for and how to check it: a representation should expose the state of a particular event, and this is checked by transfer to event-level questions it was not trained on. The second is the hypothesis you describe: that objectives which explicitly target this state transfer better. You are right that the submission did not train this objective or compare it with existing objectives. Following your comment, we conduct this comparison at pilot scale. We found no transfer advantage for contact-state training on remaining holding time.

Experimental setup: We train four matched adapters on current contact and change, relation categories, target location, or future visual features. An adapter is a small trainable layer on top of the same frozen features, and matched means that the four differ only in this training target. We then freeze each adapter and fit a holding-time readout on separate recordings (Table 1 and Appendix I.1). The readout is a small model that reads the answer from the frozen adapter. Because each adapter is frozen, the readout can use only what that adapter has already kept. This is what we mean by transfer.

Results: The prespecified readout, whose settings we fixed before the experiment, was unstable on one fold, because it divided by a standard deviation that was close to zero. A fold is one of the three splits in which one recording is held out for testing. We therefore report a post-hoc check that omitted variance scaling (this division) and used all available adaptation events, which are the labeled events used to fit the readout. In this check, mean endpoint error was 1.4 s for state training and 1.2 s for recognition under either annotator (Table 19). The endpoint error is the error, in seconds, of the predicted time left.

This comparison covers three development recordings and one held-out question. It is a first test of the hypothesis at pilot scale. As a further comparison with existing objectives, the revised §3.6 (lines 428–431) reads four frozen pretrained encoders with matched remaining-time readouts. All four had lower mean endpoint error than the training-mean prior for touching under both annotators, and only one did so for holding under both (Appendix H.2, Table 17).

Section 5 sets out the research agenda, in which the handover phase is a second, independent transfer question. A handover is one person passing an object to another.

AuthorsOpenReview ↗

Response to Reviewer o3NJ (Part 2/3)

W2: Insufficient experiment.

Response 2: Thank you for raising the sample size, and we apologize for any confusion. We here clarify that the pilot contains two results with different levels of support:

  • Recognition, measured on 40 validation videos:
    • Supplying both participants improved recognition Recall@3 by 19.4 percentage points over actor-only input (Table 2a). Recall@3 is the share of recorded relation labels ranked among the top three.
  • Timing, an exploratory pilot on eight videos:
    • All three primary 95% contrast intervals included zero (Appendix B, p. 20). A contrast is the difference in error between the recognition-trained readout and one of the three controls. An interval that includes zero means the data do not show which of the two is better.
    • On nine shared events, which are the events that both human annotators record, the duration prior also had the lowest mean error against each annotator's boundaries (Table 14). The duration prior is a baseline that sees the event's label but not the video.

We have revised §3.6 and the conclusion to limit the timing claim to the readout and training budget we tested. The changes are marked in blue at:

  • §3.2, lines 305–307: the result is limited to the tested interface and training budget, and the text states that it does not establish a separation between recognition and interaction-state understanding.
  • §3.3, lines 320–322: the recognition margin under the timing head's own inputs is described as uncertain, with its interval.
  • §3.6, lines 436–437: the limitations now open with the statement that recognition training showed no reliable remaining-time advantage with the tested readout and training budget.
  • Conclusion, lines 531–532: the same limit to the tested interface and training budget.
AuthorsOpenReview ↗

Response to Reviewer o3NJ (Part 3/3)

W3: Confusion in temporal evidence.

Response 3: Thank you for this insightful comment. We agree with both points. The pilot does not tell these explanations apart, and all of them remain plausible. We here report what we can say about each.

  • The representation or the causal head:
    • A linear probe [C1] on the top ten principal components of the trained features explained 5.3% of pooled remaining-time midpoint variance (Table 11), that is, of the variation in the time left across all scored frames. A linear probe is a linear model fitted on frozen features to check what can be read from them. Some timing information was accessible, and the estimate depends on how it was read out. This is why we limit the timing result to the way the features were read out in our test, and do not extend it to the representation itself (§3.2, lines 307–308).
  • An unobservable human intention:
    • We agree that this limits the remaining time as a metric. The paper treats it as one diagnostic among several, which can be unpredictable when the end depends on choices the past frames do not show (§3.2, lines 258–260). Following your comment, we test current contact, whose label does not depend on when the person later lets go. This takes the future choice out of the test, so what remains is the effect of the observations and the readout.
  • Temporal observations:
    • In the same test we give the readout two inputs: the current frame alone, or the mean of the 32 frames ending at the current frame.

Experimental setup: Both inputs are scored on the same 521 time points in three recordings. Of these, 141 fall inside both annotators' contact intervals. We infer 380 negatives from the two complete event records (Appendix H.4). These are time points that lie outside every possible touching interval of both annotators. The input is the frozen whole-frame DINOv2 feature. We report two metrics. Balanced accuracy is the accuracy averaged over contact and no-contact time points at a fixed threshold. AUROC measures how well the scores rank contact above no-contact time points across all thresholds.

Results: The results are reported in Table C1.

Table C1: Current-contact readout with two observation histories (%, higher is better for both metrics). Means give each recording equal weight. Revised manuscript Table 21. Best results are highlighted in bold.

Whole-frame DINOv2 inputBalanced accuracy ↑AUROC ↑
Current frame47.674.6
Mean of 32 frames ending at the current frame55.461.3

Averaging history improved fixed-threshold balanced accuracy and reduced AUROC. The two metrics moved in opposite directions, so this test does not show a clear benefit of a longer history. Because averaging also discards frame order, this test cannot isolate the effect of additional history alone.

Thank you for drawing attention to this distinction. Whether the timing error comes chiefly from the readout or from uncertainty about a participant's next decision remains open. The paper states the timing result with this limit: a stronger head or better observations could yield a positive result (§3.6, line 440).


[C1] Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics 2022.

Reviewer_rD2bOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that representation learning for physical interaction should explicitly preserve participant-based interaction states and relational changes, rather than treating them as an automatic byproduct of object perception or relation recognition. The paper introduces a transfer-oriented framework and conducts a SAM4D pilot study using frozen SAM 3 features.

Critical strengths

The pilot project is highly transparent about both its negative results and its conclusions. The matched random readouts, zero-vision and label-only controls, video-level bootstrap intervals, endpoint audits, and manual label audits make this narrow empirical conclusion more credible than a typical small-scale exploratory study.

Critical weaknesses

The paper's core thesis is not validated by the experiments it presents. The title and position argument claim that interaction understanding "requires more than object-centric representations," and the first part further hypothesizes that objectives explicitly targeting interaction states should transfer better than recognition or general prediction objectives. However, the only source objective actually trained is relation recognition, and the temporal study compares readouts trained via recognition against their random initialization, zero-vision input, and duration priors. No interaction-state objective is trained, no matched object-centric or video-prediction objectives are compared, and only a single downstream problem is considered.

Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer rD2b (Part 1/3)

Dear Reviewer rD2b,

Thanks for your valuable comments! We address your concerns below. Please kindly let us know if you have further concerns.

W1: The paper's core thesis is not validated by the experiments it presents.

Response 1: Thank you for this clear and constructive comment. Following your suggestion, we conduct the test you describe at pilot scale. Our main message is twofold:

  1. We followed your suggestion. We train an interaction-state objective, compare it with recognition, object-centric and predictive objectives under matched conditions, and add a second downstream question (Responses 1.1 to 1.4).
  2. The result supports the first part of our position. At pilot scale, replacing relation-recognition training with explicit interaction-state supervision did not improve transfer to a different event-level question. This suggests that our original negative result is not simply an artifact of training only on relation recognition. More broadly, it supports testing transfer directly: the ability to answer event-level questions should not be assumed from the training objective or from success on the training task. It has to be demonstrated on held-out event-level questions.

We also apologize for any confusion about what the paper claims. The position has two parts (Section 1, lines 087–091):

  • First part, what to aim for and how to check it: A representation should expose the state of a particular event, not only its category, and this is checked by transfer to event-level questions it was not trained on. The "more" in our title is this state (lines 077–079). An object-centric model that exposes it meets this part (Section 4, lines 453–455).
  • Second part, a hypothesis: Objectives that explicitly target this state transfer better than general objectives. The submission presented this as a hypothesis for future tests. The new experiment tests it at pilot scale and finds no advantage for the state target. The revised abstract, Section 5 and conclusion report this.

W1.1: "No interaction-state objective is trained"

Response 1.1: Thank you for pointing out this gap between what the paper advocates and what the pilot trained. Following your suggestion, we train an adapter directly on interaction state and test whether what it learns helps with a new question.

Experimental setup:

  • What is trained: An adapter, a small trainable layer on top of the frozen SAM 3 features, learns to predict current pair contact and its signed change since five frames earlier (contact gained, lost, or unchanged).
  • Why this target: When the receiver first touches a cup, the giver's holding label can stay true. The label stays the same while the interaction changes. Our hypothesis is that preserving this change will make other event questions easier to learn.
  • How it is tested: We discard the training head, freeze the adapter and fit a fresh readout for remaining holding time. The readout is a small model that reads the answer from the frozen adapter. Because the adapter is frozen, this new question has to be answered from what the adapter has already kept.

Results: The comparison with three other objectives is reported in Response 1.2 (Table B1). One limit of this pilot is that changes in contact were rare in the training data. Only 15 of 327 overlapping source windows, the 32-frame clips used for training, had different contact states at the two sampled times (Appendix I.1).

AuthorsOpenReview ↗

Response to Reviewer rD2b (Part 2/3)

W1.2: "no matched object-centric or video-prediction objectives are compared"

Response 1.2: Thank you for asking for a matched comparison. Following your suggestion, we conduct one with four training objectives.

Experimental setup: Matched means that the four arms differ only in what they are trained to predict. The four arms (one adapter per objective) share frozen SAM 3 features, participant masks, three source recordings, initialization, 68 adapter parameters and 200 updates. We vary the target as specified in manuscript Table 1:

  • Pair contact and change: the interaction-state objective of Response 1.1.
  • Relation recognition: the recognition objective. This arm uses the recorded PVSG [B1] predicates (relation labels) caressing and playing with.
  • Target-box location: an object-centric objective, which predicts where the target object is in the image.
  • Future visual features: a predictive objective, which predicts a summary of the visual features of the next 32 frames.

We then freeze each adapter and fit a new readout for remaining holding time on three other development recordings. The labeled events used to fit this readout are the adaptation events. The error is the endpoint MAE: the mean absolute error, in seconds, of the predicted time left, averaged over the two ends of the annotated end range.

A numerical problem and the post-hoc check: The prespecified readout, whose settings we fixed before the experiment, divides each input coordinate by its standard deviation on the training recordings. It became unstable on one fold when it divided by a near-zero training standard deviation: one output of the contact-state adapter was almost constant on the training recordings of that fold. A fold is one of the three splits in which one recording is held out for testing. A post-hoc sensitivity check omitted that division and kept the splits, centering and ridge coefficient (the strength of regularization) fixed. Because we chose this check after seeing the results, revised manuscript Table 19 gives both readouts.

Results: The results of the post-hoc check are reported in Table B1.

Table B1: Transfer to remaining holding time (endpoint MAE in seconds ↓). Post-hoc check using all available adaptation events, two or three per fold. Means weight three held-out development recordings equally. Revised manuscript Table 19 gives both readouts.

Source objective or controlAnnotator AAnnotator B
Pair contact and change1.41.4
Relation recognition1.21.2
Target-box location1.41.3
Future visual features1.31.2
Untrained adapter1.21.1
Training-label mean1.41.3

The last two rows are controls. The untrained adapter is the same adapter without training, and the training-label mean predicts the average training label without visual input.

Existing pretrained encoders: In addition, the revised §3.6 (lines 428–431) compares four frozen pretrained encoders under matched remaining-time readouts: VideoMAE V2 [B2] and V-JEPA 2 [B3], which are pretrained by predicting missing parts of videos, DINOv2 [B4] and SAM 3 [B5]. All four had lower mean endpoint error than the training-mean prior for touching under both annotators. Only V-JEPA 2 did so for holding under both (Appendix H.2, Table 17).


[B1] Panoptic Video Scene Graph Generation. CVPR 2023.

[B2] VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking. CVPR 2023.

[B3] V-JEPA 2: Self-supervised Video Models Enable Understanding, Prediction and Planning. arXiv 2025.

[B4] DINOv2: Learning Robust Visual Features without Supervision. arXiv 2023.

[B5] SAM 3: Segment Anything with Concepts. ICLR 2026.

AuthorsOpenReview ↗

Response to Reviewer rD2b (Part 3/3)

W1.3: "the temporal study compares readouts trained via recognition against their random initialization, zero-vision input, and duration priors"

Response 1.3: Thank you for the insightful comment, and we apologize for any confusion about what these controls can show. We here clarify what the original controls test, why they are not a test of reuse, and what we add.

  • What the original controls test:
    • The original random readout, zero visual input and duration prior test what recognition training and visual input contribute to timing. Each control removes one ingredient. The random readout removes recognition training, the zero visual input removes the video, and the duration prior keeps only the event's label.
  • Why they are not a test of reuse:
    • Reuse means that a new question can be learned from fewer labels. The original controls all receive the same number of labels, so they cannot show a saving.
  • What we add:
    • To test reuse, we also limited adaptation to one labeled event. In the same post-hoc check, the contact-state adapter and the training-label mean both round to 1.4 s for A and 1.3 s for B (Table 19, one-event columns). With one labeled event, the adapter trained on contact state was therefore no better than a baseline that does not look at the video.

W1.4: "only a single downstream problem is considered"

Response 1.4: Thank you for encouraging us to look beyond remaining time. Following your suggestion, we evaluate a second downstream question: are the participants touching now? We choose it because its answer depends only on the present, whereas the remaining time depends on what the person does next.

Experimental setup: In a separate DINOv2 experiment, we train on absolute or relative 2D geometry in eight videos and evaluate current contact on three other videos. Absolute geometry means where the two participants are in the image and how they have moved. Relative geometry means how far apart they are and how that distance has changed. The control is a fixed random projection of the same features, which receives no geometry training. The metric is the Brier score, the mean squared difference between the predicted probability of contact and the label.

Results: Mean Brier scores (×100, lower is better) were 22.3 for absolute geometry, 23.5 for relative geometry and 18.7 for a fixed random projection (Table 20). The random projection had the best mean score. On this test, training on 2D geometry therefore did not make current contact easier to read.

Reviewer_ZQApOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that learning should target the changing interaction state between specific participants. It proposes evaluating transfer to new event-level questions under controlled adaptation budgets. A SAM4D pilot improves recognition through participant conditioning. However, it shows no reliable remaining-time prediction gain on 15 events from eight videos. The paper presents matched objective comparisons as future tests.

Critical strengths
  1. A useful and falsifiable position. The paper separates access to interaction state from the hypothesis that explicit state objectives improve transfer. It treats object-centric dynamics and general pretraining as serious alternatives (Sections 4-5). It also states which outcomes would weaken its hypothesis.

  2. Careful empirical reporting. The pilot includes matched temporal heads, nonvisual controls, a duration prior, video-level uncertainty, and independent human annotation. The same-prefix recognition replay addresses an important input mismatch. These controls make the limitations informative.

Critical weaknesses
  1. The pilot's interpretation is occasionally too strong. Page 8 claims that eight videos establish a separation of capabilities. Yet the same-prefix recognition margin and all three primary temporal contrasts have intervals crossing zero (Appendices B-C). Remaining time may also depend on unobserved future choices. The linear probe extracts some timing information (Table 10). The result shows no demonstrated temporal advantage at the tested interface and budget. The stronger wording should follow Table 12's caution.

  2. The proposed transfer test needs a more precise operational example. Section 5 requires unseen questions, but leaves the exact supervision unspecified. Progress and next-state targets may overlap with remaining-time and handover questions. A concrete training/test specification would clarify which supervision is excluded. It would also clarify the contribution beyond existing relational dynamics evaluations.

Questions for authors
  1. Clarification: What specific interaction-state target would be trained, and which two question types would be held out? What supervision would be excluded during representation training?

  2. Robustness check: Do the temporal comparisons persist when saved predictions are scored against each human annotator on common observable support? The nine-event analysis still uses automatic boundaries (Appendix D). This check would strengthen confidence in the pilot without requiring a new model.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZQAp (Part 1/3)

Dear Reviewer ZQAp,

Thanks for your valuable comments, and for reading the paper so closely! We are glad that you found the position useful and falsifiable. In short, we have revised the pilot's wording, made the test in Section 5 concrete (W2 and Q1), and run the robustness check you propose (Q2). Our main message from that check is twofold:

  1. The original timing result is not an artifact of the automatic boundaries. On the nine events that both human annotators record, it persists under each annotator's own boundaries (Response 3).
  2. More broadly, this supports testing transfer directly. The ability to answer an event-level question should not be assumed from recognition performance. It has to be demonstrated on that question.

We address your concerns below. Please kindly let us know if you have further concerns.

W1: The pilot's interpretation is occasionally too strong.

Response 1: Thank you for the careful comment, and we apologize for any confusion about the original wording.

  • Intervals that include zero: The independent units are the eight videos. All three primary timing contrasts, which compare the trained readout with each of the three controls, have 95% intervals that include zero (Appendix B). Even the same-prefix recognition margin over the frequency prior had an interval that included zero (Table 6). In that check, the recognition classifier sees exactly the frames that the time-left predictor sees, and the frequency prior ranks labels only by how often they occur in the training data.
  • Unobserved future choices: How long a person keeps holding an object can depend on a choice that the past frames do not show. Our human labels cannot tell us whether the end was predictable. Our annotators first saw each clip during annotation and could inspect its ending. Someone predicting from a prefix still cannot see when the person will choose to let go (Appendix D).
  • Timing information in the features: As you note, the linear probe extracts some timing information from the trained features (Table 10 in the submission, now Table 11). The pilot therefore does not show that this information is absent.

Following your suggestion, the manuscript has been revised. The changes are marked in blue at:

  • §3.2, lines 305–307: the result is limited to the tested interface and training budget, and the text states that it does not establish a separation between recognition and interaction-state understanding.
  • §3.3, lines 320–322: the same-prefix recognition margin is described as uncertain, with its interval.
  • §3.6, lines 436–437: the statement above replaces the earlier claim of a separation.
  • Conclusion, lines 531–532: the same limit to the tested interface and training budget.
AuthorsOpenReview ↗

Response to Reviewer ZQAp (Part 2/3)

W2: The proposed transfer test needs a more precise operational example.

Q1: What specific interaction-state target would be trained, and which two question types would be held out? What supervision would be excluded during representation training?

Response 2: Thank you for the constructive suggestion, and we apologize that the original Section 5 was not specific enough. Following your suggestion, Section 5 now gives a concrete training/test specification (lines 486–506), and we have run a first instance of it (lines 479–485). You are right that progress and next-state targets can overlap with the remaining-time and handover questions, because such labels can encode the answers. The rule behind the specification is therefore simple: a question counts as held out only if no training label already contains its answer. Specifically:

  • Train:
    • Whether the specified pair is touching now and the signed change in contact since five frames earlier (contact gained, lost, or unchanged). This target is local to the present and the past, so it contains no label about how the event will end. We trained this target and froze the adapter, a small trainable layer on top of the frozen features, before fitting a holding-time readout, a small model that reads the remaining holding time from the frozen adapter (Tables 1 and 19).
  • Hold out:
    • How much longer will contact last? and Has the handover begun? A handover is one person passing an object to another. The second question requires visible transfer of support or control. A first touch alone cannot establish that transfer.
  • Exclude:
    • Remaining time, normalized progress (the fraction of the event that has already passed), future endpoints, handover phase, support-transfer labels and equivalent encodings, that is, any label from which these can be computed. They are excluded from representation fitting and selection, which means both training the representation and choosing which trained version to keep. This includes complete labeled trajectories that reveal an ending. Each of these labels would give away the answer to one of the two held-out questions. If a question turns out to be equivalent to a training label, we report it as a seen task rather than as transfer.
    • Progress and next-state labels can still be training targets in other experiments, but the questions they encode are then no longer called held out (lines 509–511).
    • The predictive baseline may still use future visual features as training targets. The restriction concerns what the representation receives as input at test time, and it is not a ban on every training target that lies in the future.

Regarding the contribution beyond existing relational dynamics evaluations, SlotFormer [A1] shows that object dynamics can support downstream question answering and planning. Building on this, our proposed test measures how much additional supervision each frozen representation needs on both event questions, with the backbone, recordings, observed history, capacity and training compute matched across objectives. The cost of source labels is reported separately, because annotating the state is itself a cost that the comparison has to count.


[A1] SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models. ICLR 2023.

AuthorsOpenReview ↗

Response to Reviewer ZQAp (Part 3/3)

Q2: Do the temporal comparisons persist when saved predictions are scored against each human annotator on common observable support?

Response 3: Thank you for this concrete check. You are right that the nine-event analysis in the submission still used the automatic boundaries. Following your suggestion, we rescore the fixed predictions against each annotator's own boundaries, without training a new model. The answer is yes for the principal timing conclusion.

Experimental setup: We keep every model and every saved prediction fixed and change only the labels used for scoring, so any change in the result comes from the labels alone. All methods are scored on the same 115 event–frame contexts from nine events in five videos, where a context is one frame of one event. These contexts form the common observable support: the frames that both annotators place inside the event and mark as judgeable. Each annotator marks the end of an event as a short range of frames, the endpoint bracket. An error of zero means the prediction falls inside that annotator's endpoint bracket. In the first three methods below, the same small head predicts the time left, and only the features it receives differ:

  • Recognition-trained readout: the head receives features from SAM4D, our small readout trained on frozen features to recognize relations.
  • Randomly initialized readout: the head receives features from the same readout before any training. It shows what recognition training adds.
  • Zero visual input: the visual features are set to zero. It shows what the video adds.
  • Duration prior: a baseline that sees the event's label but not the video. It predicts the typical time left for that label in the training videos.

Results: The results are reported in Table A1.

Table A1: Fixed-prediction interval error by human annotator (seconds ↓, videos weighted equally). Revised manuscript Table 14 also gives the automatic labels. Best results are highlighted in bold.

MethodAnnotator AAnnotator B
Recognition-trained readout1.00.8
Randomly initialized readout1.21.1
Zero visual input1.00.9
Duration prior0.80.7
  • Trained versus random, and trained versus zero visual input: For both annotators, the paired 95% video-bootstrap intervals included zero. These intervals are obtained by resampling the videos, and an interval that includes zero means the data do not show which of the two methods is better.
  • Trained minus prior: The interval was above zero for A and included zero for B (Table 15). The prior ranked first under both annotators, although its margin over the trained readout was more uncertain for B.

Empirically, under each annotator's boundaries the recognition-trained readout still shows no reliable advantage over the controls, and the duration prior still has the lowest error. On the nine events that both annotators record, the original timing result is therefore not an artifact of the automatic boundaries.

AuthorsOpenReview ↗

We would love to hear your voice

Dear Reviewer ZQAp,

The discussion period is entering the end, and we did not want it to close without saying this: we are really happy to have the chance to discuss our work openly with you here. You read the paper more closely than we could have hoped, and it is a better paper because of it. Thank you!

To make it easy to jump in, here is the short version of what we did with your comments:

  • W1 (wording too strong): the pilot's claim now follows your wording: no reliable remaining-time advantage at the tested readout and budget.
  • W2 and Q1 (make the test concrete): Section 5 now spells out what is trained, what is held out, and what is excluded.
  • Q2 (robustness check): we rescored the saved predictions against each human annotator, with no new model.

The key point: your Q2 check came back with a yes. On the nine events that both annotators record, the timing result holds under each annotator's own boundaries, so it is not an artifact of the automatic boundaries.

We would love to hear what you think. Does the new check answer your question? Is the Section 5 specification concrete enough now? Whether you agree or still see a gap, we would be glad to talk it through while the discussion is open. Even a short note would make our day :)

Best regards,

Submission 20857 Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers,

We appreciate all the reviewers for their insightful and constructive reviews of our manuscript. We are glad that the reviewers found that:

  • The paper raises a meaningful problem (Reviewer o3NJ) and states a useful and falsifiable position (Reviewer ZQAp).
  • The pilot is reported carefully and transparently, with matched controls and human annotation audits (Reviewer ZQAp, rD2b, and o3NJ).
  • The paper treats object-centric dynamics and general pretraining as serious alternatives (Reviewer ZQAp).

Based on all the reviews, here we provide a general response to the concerns raised by multiple reviewers. The individual responses are commented on below each review.

Our position in one example. As a receiver first touches a cup, the giver may still be holding it. The relation label stays true even as the interaction changes. Our position is that a representation should preserve such changes between the specified participants, and that this should be checked by transfer to new questions about the event (first part). Whether explicitly supervising these changes helps a frozen representation answer new questions with fewer labels is a hypothesis (second part).

Main message of the new results.

  1. The original timing result is not an artifact of the automatic boundaries. On the nine events that both human annotators record, it persists when the saved predictions are scored against each annotator's own boundaries.
  2. Changing the training objective did not change the picture. At pilot scale, replacing relation-recognition training with explicit interaction-state supervision did not improve transfer to a different event-level question. This suggests that the original negative result is not simply an artifact of training only on relation recognition.

Both results support testing transfer directly, which is the first part of our position: the ability to answer event-level questions should not be assumed from the training objective or from success on the training task. It has to be demonstrated on held-out questions. The second part remains a hypothesis: this pilot found no advantage for the state target, as the revised abstract, Section 5 and conclusion report.

  • Regarding questions about experiments, we have addressed the concern as follows:
    • We narrow the timing claim to the tested interface and training budget, because eight videos are too few to show that recognizing an interaction and predicting when it ends are separate abilities (§3.2, §3.3, §3.6 and conclusion; lines 305–307, 320–322, 436–437 and 531–532). (For Reviewer ZQAp and o3NJ)
    • Notably, we follow Reviewer ZQAp's suggestion to rescore the fixed predictions against each human annotator. The duration prior, a baseline that sees the event's label but not the video, had the lowest mean error under both annotators on nine shared events (Tables 14–15). (For Reviewer ZQAp and o3NJ)
    • We compare four matched training objectives (contact state, relation categories, target location, future visual features) with small adapters on the same frozen features, and test transfer to holding time on three other development recordings (Tables 1 and 19). The readout specified in advance was unstable on one held-out recording, because it divided by a near-zero standard deviation. In a post-hoc check without this division, contact-state training gave higher mean error than recognition under both annotators. (For Reviewer rD2b and o3NJ)
    • We add a question about the present: is the pair touching now? Its answer does not depend on what a person decides to do next. Extending the input from one frame to a 32-frame mean improved balanced accuracy but lowered AUROC on 521 shared current-contact time points from three recordings (Tables 20–21). (For Reviewer rD2b and o3NJ)
  • Regarding questions about how the position can be tested, we have addressed the concern as follows:
    • Section 5 now specifies the training target, two held-out questions, and the supervision excluded during representation learning (lines 486–506). (For Reviewer ZQAp)
    • Section 5 sets out the research agenda, in which the handover phase is a second, independent transfer question. That test requires independent recordings with reviewed support-transfer labels, and the position of the current paper does not rely on it.

We have also uploaded a revised version of our submission according to the suggestions provided by reviewers, with major changes highlighted in blue. We look forward to further feedback and discussion. Please feel free to let us know if further details or explanations are helpful.

Once again, we thank all reviewers for their thoughtful feedback and valuable guidance in improving our work.

Best regards,

Submission 20857 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.