acceptodds
Withdrawn from ICLR 2027

What Happens Next? Future-Based Reasoning for Streaming Human–Object Interaction Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Many video human-object interaction predicates are defined by the change they induce rather than the configuration they display, so a person beginning to push a box and one beginning to pull it look identical until the box moves. Offline models widen the observation window until the consequence appears, an option unavailable to systems that must interpret an interaction while it unfolds. We study causal streaming human-object interaction detection under a bounded-lag protocol in which the predicate may be revised once after a fixed lag while tubes, identities, and temporal boundaries stay frozen, so no gain can come from delayed localization and every method sees the same later frames. Our starting point is a measurement. A frozen latent video world model rolled out without modification produces one future shared by every candidate and carries almost no information about which predicate was applied, because the regressed future averages over what the candidates would each do. Conditioned on one candidate at a time, the same frozen model separates confusable predicates, but only inside a bounded horizon band. RIVET turns the frozen model into a bank of predicate-conditioned simulators, anchors the branches of unrealized predicates to real continuations of comparable scenes, scores each simulation against the continuation that actually arrives, and returns the prior unchanged when the candidates predicted indistinguishable futures. Learn-then-test calibration bounds per-video degradation without assuming the world model, the simulator, or the critic is correct. On streaming VidOR and a temporally confusable subset of Something-Else, RIVET improves relation detection mAP from 11.98 to 12.22 against a recognizer given the same extended window, with gains rising monotonically with measured separation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video human-object interaction detection, streaming / causal video understanding, video world models, counterfactual reasoning, conformal prediction, learn-then-test calibration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_TqR3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work introduces a new task of Streaming Human-Object Interaction detection, which adopts a bounded-lag protocol to revise interaction recognition. The weakest aspect of the paper is arguably its ablation study, which makes it difficult for readers to understand whether the proposed revision mechanism actually benefits interaction recognition.

Critical strengths

This work introduces a new task of Streaming Human-Object Interaction detection, which adopts a bounded-lag protocol to revise interaction recognition. The motivation and visualizations are quite clear.

Critical weaknesses

Many experimental details are insufficiently specified, making the results difficult to reproduce. The connection between the experimental setup and the stated motivation is also not clearly articulated.

Questions for authors

1 In Table 2, the “Without sep. gate” setting may serve as an ablation of the revision mechanism. However, its impact is minimal, with an mAP drop of only −0.001, which appears inconsistent with the paper’s main motivation. It remains unclear whether this limited effect stems from the experimental setup rather than from the revision mechanism itself.

2 We further examine the design of this gate, which may correspond to the “Gate4 scoring” described in Section 3.2. This appears to be a conditional piecewise function involving a relatively complex four-condition mechanism. However, the function is not fully specified, and it is unclear whether it contains learnable parameters, which may affect the gate’s experimental behavior and makes the presentation unfriendly to readers.

3 The correspondence between Table 2 and Figure 2 is also poorly aligned. For example, the “Retrieval-Based Supervision” component shown in Figure 2 does not have a clearly identifiable technical implementation in the method description.

4 The experiments use a Gaussian Mixture Model to learn a prior, but the motivation for this choice is not well justified. It is unclear why other distributions were not considered. The evidence learning component therefore appears to lack sufficient theoretical depth. Moreover, the parameter sensitivity of the GMM is only examined in the single-Gaussian case, making it difficult to assess how sensitive the model is to GMM hyperparameters.

5 The model employs multiple loss terms, including L_sim and L_ret. However, the paper does not provide a comprehensive ablation study over the loss functions, and the loss hyperparameters are not clearly specified, which hinders reproducibility.

6 Because the paper adopts a bounded-lag protocol, many details regarding temporal partitioning and data splitting are not clearly described, further complicating experimental reproduction.

Reviewer_CwyyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RIVET, a framework for streaming human–object interaction detection under a bounded-lag prediction and revision protocol. The method uses a pretrained latent video model to predict predicate-conditioned future evidence for candidate human–object interactions. Once the actual future observations become available, a matching module compares predicted and observed evidence to refine predicate scores. The framework further incorporates retrieval-based supervision, separation-aware gating, and risk calibration to constrain potentially harmful revisions.

Critical strengths
  • The paper addresses streaming human–object interaction detection with bounded-lag predicate revision, a relevant setting for online video understanding.
  • The paper presents its prediction, matching, and calibration components clearly and includes several controlled ablations and uncertainty analyses.
Critical weaknesses
  • The approach mainly integrates established ideas from future prediction, conditional generative modeling, retrieval-based supervision, residual score correction, and risk calibration. Prior works such as Anticipative Video Transformer (ICCV 2021), Towards Scene Graph Anticipation (ECCV 2024), and Towards Unbiased and Robust Spatio-temporal Scene Graph Generation and Anticipation (CVPR 2025) have already investigated future-oriented video understanding and relation anticipation. Although bounded-lag predicate revision provides a different task formulation, the paper does not sufficiently establish a substantial methodological advance beyond combining these existing techniques.
  • The ablation results indicate that permuting candidate-specific future evidence preserves approximately 91% of the full model's gain over the action expert. Similarly, removing retrieval supervision reduces mAP by only 0.026 points, while always opening the separation gate changes mAP by just 0.001 points. These findings suggest that the proposed components provide limited measurable contributions, making it difficult to attribute the improvements to the claimed future-conditioned reasoning mechanism.
  • The paper is motivated by interactions that cannot be distinguished from initial observations but become identifiable through their future consequences. However, the improvements do not concentrate on consequence-defined predicates, and the reported AP on the VidOR CD-15 subset decreases. This weakens the empirical connection between the motivating failure mode and the proposed solution.
  • The figures in the paper may be used AI-generated tool to make them. It is quite sad about this due to the research by themselves that coud not be drawn.
Questions for authors
  • How does the proposed approach differ substantively from prior work on action and scene graph anticipation, particularly in terms of the underlying predictive mechanism?
  • Why does the proposed method fail to demonstrate clear improvements on predicates whose identities depend on future consequences, despite these being the central motivation?
  • How frequently does the separation gate suppress revisions, and what proportion of proposals receive meaningful score corrections?
Reviewer_mD1wOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies streaming human-object interaction detection with one predicate revision after a fixed delay. The proposals, identities, and temporal intervals remain fixed, so revision only changes the predicate scores. RIVET predicts a future evidence distribution for each candidate predicate and compares it with the continuation that later arrives. It also uses retrieved transitions, a separation gate, and risk calibration. Experiments are reported on VidOR, VidVRD, and an eligible-frame subset of Action Genome. The main positive result is a small full-video mAP improvement on VidOR.

Critical strengths
  1. The bounded-lag revision problem is useful. Freezing localization while revising predicates provides a clear way to study whether later observations help interpret the same interaction.
  2. The paper includes relevant negative controls, especially permuted candidate evidence and an always-open separation gate. These controls are important for checking whether the improvement comes from the proposed future-matching mechanism.
Critical weaknesses
  1. The main streaming claim is not established. Table 1(c) does not report RIVET under the matched observation budget. Comparing a prefix recognizer with a delayed recognizer only shows that seeing more frames helps.
  2. Permuted candidate evidence retains about 91% of the full improvement over the action expert. The current gain therefore gives little evidence for the need for candidate-specific futures and separation gating.
  3. The results do not support the consequence-defined interaction motivation. CD-15 AP decreases by 0.25 points on VidOR, and Action Genome shows almost no change. The overall VidOR gain is only about 0.24 points, while both recall measures decrease slightly.
  4. Table 1(a) gives 24.65 mAP for the expert and 25.64 for native-score RIVET, a gain of 0.99 points. Section 4.2 reports a gain of 0.19 points.
  5. Several promised experiments cannot be checked. The paper refers to Appendix A for consequence-defined subsets, additional controls, uncertainty estimates, and failure cases, but the supplied PDF contains no appendix. The claimed finite useful prediction horizon and label-efficiency benefit are also not demonstrated by the available results.
  6. The visual analysis does not examine the proposed future model directly. Figure 3 shows two development examples and revised scores, but not the predicted candidate distributions or their agreement with the observed continuation. Figure 2 is also too dense to explain the full procedure clearly. Failure cases and separation-versus-revision plots would be more informative than additional conceptual diagrams.
Questions for authors
  1. Why does permuting candidate evidence preserve most of the improvement? Which experiment establishes a benefit from pretrained, predicate-conditioned dynamics over a generic residual reranker?
  2. What are the calibration sample size, grid size, tolerances, selected operating point, revision coverage, and measured degradation risks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.