- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper argues that it is difficult to adequately understand interaction states that change over time using only existing object-centric representations and interaction recognition, and that these should be treated as explicit learning targets. To support this, we present experimental results showing that participant conditioning improves relation recognition performance using SAM4D based on SAM 3, but the trained representations do not demonstrate a consistent advantage over the baseline in predicting the remaining-time of interactions.
Critical strengths
- This paper raises a meaningful problem by distinguishing interaction recognition from interaction-state understanding and emphasizing the importance of event-level reasoning and downstream transfer.
- This paper strive to ensure the reliability of experimental results through various controls and analyses, such as random readout, non-visual baselines, and human annotation audits.
Critical weaknesses
- Unverified Central hypothesis : The main argument is that explicit interaction-state learning provides better transfer, but the study did not actually learn the objective or compare it with existing objectives. Therefore, there is a lack of direct experimental evidence supporting the paper's main argument.
- Insufficient experiment : The temporal evaluation consisted of only 8 videos and 15 events, and the major performance differences were not statistically significant. Therefore, it is difficult to conclude that recognition and interaction-state understanding are actually separate.
- Confusion in temporal evidence : It is difficult to distinguish whether the failure of remaining-time prediction is due to the limitations of the representation or the limited causal head and temporal observations. In particular, event termination can rely on an unobservable human intention, which limits its use as an evaluation metric for understanding interactions.
Questions for authors
see weakness
6 comments
Response to Reviewer o3NJ (Part 2/3)
W2: Insufficient experiment.
Response 2: Thank you for raising the sample size, and we apologize for any confusion. We here clarify that the pilot contains two results with different levels of support:
- Recognition, measured on 40 validation videos:
- Supplying both participants improved recognition Recall@3 by 19.4 percentage points over actor-only input (Table 2a). Recall@3 is the share of recorded relation labels ranked among the top three.
- Timing, an exploratory pilot on eight videos:
- All three primary 95% contrast intervals included zero (Appendix B, p. 20). A contrast is the difference in error between the recognition-trained readout and one of the three controls. An interval that includes zero means the data do not show which of the two is better.
- On nine shared events, which are the events that both human annotators record, the duration prior also had the lowest mean error against each annotator's boundaries (Table 14). The duration prior is a baseline that sees the event's label but not the video.
We have revised §3.6 and the conclusion to limit the timing claim to the readout and training budget we tested. The changes are marked in blue at:
- §3.2, lines 305–307: the result is limited to the tested interface and training budget, and the text states that it does not establish a separation between recognition and interaction-state understanding.
- §3.3, lines 320–322: the recognition margin under the timing head's own inputs is described as uncertain, with its interval.
- §3.6, lines 436–437: the limitations now open with the statement that recognition training showed no reliable remaining-time advantage with the tested readout and training budget.
- Conclusion, lines 531–532: the same limit to the tested interface and training budget.
Response to Reviewer o3NJ (Part 3/3)
W3: Confusion in temporal evidence.
Response 3: Thank you for this insightful comment. We agree with both points. The pilot does not tell these explanations apart, and all of them remain plausible. We here report what we can say about each.
- The representation or the causal head:
- A linear probe [C1] on the top ten principal components of the trained features explained 5.3% of pooled remaining-time midpoint variance (Table 11), that is, of the variation in the time left across all scored frames. A linear probe is a linear model fitted on frozen features to check what can be read from them. Some timing information was accessible, and the estimate depends on how it was read out. This is why we limit the timing result to the way the features were read out in our test, and do not extend it to the representation itself (§3.2, lines 307–308).
- An unobservable human intention:
- We agree that this limits the remaining time as a metric. The paper treats it as one diagnostic among several, which can be unpredictable when the end depends on choices the past frames do not show (§3.2, lines 258–260). Following your comment, we test current contact, whose label does not depend on when the person later lets go. This takes the future choice out of the test, so what remains is the effect of the observations and the readout.
- Temporal observations:
- In the same test we give the readout two inputs: the current frame alone, or the mean of the 32 frames ending at the current frame.
Experimental setup: Both inputs are scored on the same 521 time points in three recordings. Of these, 141 fall inside both annotators' contact intervals. We infer 380 negatives from the two complete event records (Appendix H.4). These are time points that lie outside every possible touching interval of both annotators. The input is the frozen whole-frame DINOv2 feature. We report two metrics. Balanced accuracy is the accuracy averaged over contact and no-contact time points at a fixed threshold. AUROC measures how well the scores rank contact above no-contact time points across all thresholds.
Results: The results are reported in Table C1.
Table C1: Current-contact readout with two observation histories (%, higher is better for both metrics). Means give each recording equal weight. Revised manuscript Table 21. Best results are highlighted in bold.
| Whole-frame DINOv2 input | Balanced accuracy ↑ | AUROC ↑ |
|---|---|---|
| Current frame | 47.6 | 74.6 |
| Mean of 32 frames ending at the current frame | 55.4 | 61.3 |
Averaging history improved fixed-threshold balanced accuracy and reduced AUROC. The two metrics moved in opposite directions, so this test does not show a clear benefit of a longer history. Because averaging also discards frame order, this test cannot isolate the effect of additional history alone.
Thank you for drawing attention to this distinction. Whether the timing error comes chiefly from the readout or from uncertainty about a participant's next decision remains open. The paper states the timing result with this limit: a stronger head or better observations could yield a positive result (§3.6, line 440).
[C1] Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics 2022.
Response to Reviewer o3NJ (Part 1/3)
Dear Reviewer
o3NJ,Thanks for your valuable comments! We are glad that you found the problem meaningful. Our main message is twofold:
We address the weaknesses below. Please kindly let us know if you have further concerns.
Response 1: Thank you for the valuable feedback on our central hypothesis, and we apologize for any confusion about the main argument. We here clarify that the position has two parts (Section 1, lines 087–091). The first says what to aim for and how to check it: a representation should expose the state of a particular event, and this is checked by transfer to event-level questions it was not trained on. The second is the hypothesis you describe: that objectives which explicitly target this state transfer better. You are right that the submission did not train this objective or compare it with existing objectives. Following your comment, we conduct this comparison at pilot scale. We found no transfer advantage for contact-state training on remaining holding time.
Experimental setup: We train four matched adapters on current contact and change, relation categories, target location, or future visual features. An adapter is a small trainable layer on top of the same frozen features, and matched means that the four differ only in this training target. We then freeze each adapter and fit a holding-time readout on separate recordings (Table 1 and Appendix I.1). The readout is a small model that reads the answer from the frozen adapter. Because each adapter is frozen, the readout can use only what that adapter has already kept. This is what we mean by transfer.
Results: The prespecified readout, whose settings we fixed before the experiment, was unstable on one fold, because it divided by a standard deviation that was close to zero. A fold is one of the three splits in which one recording is held out for testing. We therefore report a post-hoc check that omitted variance scaling (this division) and used all available adaptation events, which are the labeled events used to fit the readout. In this check, mean endpoint error was 1.4 s for state training and 1.2 s for recognition under either annotator (Table 19). The endpoint error is the error, in seconds, of the predicted time left.
This comparison covers three development recordings and one held-out question. It is a first test of the hypothesis at pilot scale. As a further comparison with existing objectives, the revised §3.6 (lines 428–431) reads four frozen pretrained encoders with matched remaining-time readouts. All four had lower mean endpoint error than the training-mean prior for touching under both annotators, and only one did so for holding under both (Appendix H.2, Table 17).
Section 5 sets out the research agenda, in which the handover phase is a second, independent transfer question. A handover is one person passing an object to another.