What Happens Next? Future-Based Reasoning for Streaming Human–Object Interaction Detection

Many video human-object interaction predicates are defined by the change they induce rather than the configuration they display, so a person beginning to push a box and one beginning to pull it look identical until the box moves. Offline models widen the observation window until the consequence appears, an option unavailable to systems that must interpret an interaction while it unfolds. We study causal streaming human-object interaction detection under a bounded-lag protocol in which the predicate may be revised once after a fixed lag while tubes, identities, and temporal boundaries stay frozen, so no gain can come from delayed localization and every method sees the same later frames. Our starting point is a measurement. A frozen latent video world model rolled out without modification produces one future shared by every candidate and carries almost no information about which predicate was applied, because the regressed future averages over what the candidates would each do. Conditioned on one candidate at a time, the same frozen model separates confusable predicates, but only inside a bounded horizon band. RIVET turns the frozen model into a bank of predicate-conditioned simulators, anchors the branches of unrealized predicates to real continuations of comparable scenes, scores each simulation against the continuation that actually arrives, and returns the prior unchanged when the candidates predicted indistinguishable futures. Learn-then-test calibration bounds per-video degradation without assuming the world model, the simulator, or the critic is correct. On streaming VidOR and a temporally confusable subset of Something-Else, RIVET improves relation detection mAP from 11.98 to 12.22 against a recognizer given the same extended window, with gains rising monotonically with measured separation.