- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces TimeActBench, a benchmark for evaluating whether real-time interactive models can determine, under causal, streaming observations, whether to act and when to execute an action. The benchmark comprises 196 instances across seven task categories spanning audio, video, and audiovisual modalities. It evaluates 15 models alongside human participants on both action correctness and temporal precision. The study reports a significant performance gap between models and humans and notes that low timing error on matched actions can coexist with frequent erroneous triggers.
Critical strengths
-
The research problem is highly relevant to real-time interactive systems. By shifting the focus from retrospective temporal understanding to action selection and timing as events unfold, the work addresses practical requirements of these systems.
-
The task design covers diverse modalities and temporal constraints. The evaluation considers both action correctness and timing accuracy. Human baselines help characterize task difficulty, while the supplementary code enables verification of the evaluation implementation.
Critical weaknesses
-
Differences in observation delivery may confound the attribution of results to model capabilities. In the supplementary code, the Qwen multimodal adapter waits for a full response cycle to complete after committing the input and stops sending subsequent audiovisual input during this period. The evaluation clock continues to advance, after which the system catches up by transmitting the backlog. In contrast, Gemini's media-streaming pipeline does not include this waiting mechanism. Different models may therefore receive observations at different times. Delayed responses, missed detections, or incorrect actions could arise, at least in part, from delayed observations or the deferred delivery of past states, rather than from a model's inability to make correct decisions given sufficient information. The authors should clarify how the supplementary implementation relates to the experiments reported in the paper and quantify the extent to which these differences in observation delivery affect the reported results.
-
The backlog issue is compounded by uncertainty about the temporal information available to models. Although the video-streaming functions for both Gemini and Qwen accept raw frame timestamps as parameters, the actual requests transmit only JPEG content without explicitly including the associated timestamps. Timestamps are not mandatory inputs for all real-time tasks. However, when input is blocked and subsequently delivered through a catch-up mechanism, the original video timestamp differs from the time at which the image arrives. The authors must clarify how the model identifies or reconstructs the relationship between these two time references. Otherwise, metrics computed against the original video timeline may attribute errors caused by stale observations to deficiencies in the model's temporal reasoning.
-
The experiments do not sufficiently rule out protocol-related influences on the main conclusions. Appendix I acknowledges that the reported timing reflects the entire deployed system. However, these protocol-related factors may affect not only Mean Absolute Error (MAE) but also action accuracy by altering the observations available to the model. Round-Trip Time (RTT) measurements cannot disentangle the contributions of client-side waiting, input backlogs, and submission mechanisms. The supplementary multimodal configurations also specify 1 FPS video input and a one-second commit interval, making it essential to verify whether task-relevant cues arrive before a response is required. The paper lacks analyses of actual input latency, the observability of target events, and sensitivity to scheduling strategies. It therefore remains difficult to determine the proportion of reported failures attributable to the model itself versus the evaluation framework.
Questions for authors
-
Did all experiments use the waiting, subsequent catch-up delivery, 1 FPS sampling, and one-second submission settings described in the supplementary code? Please provide representative execution traces showing the original event timestamp, the actual transmission time, and the time at which an action was received, together with input-latency measurements and the frequency with which target cues failed to arrive on time.
-
What temporal information is actually available to the model? When image requests do not explicitly include the original frame timestamp, how are backlogged observations handled, and how is the relationship between the temporal cues available to the model and the evaluation timeline made explicit?
-
How does the current scheduling strategy compare with an alternative supported by the same model's API? On a representative set of tasks, could you compare the current strategy with an alternative such as skipping stale observations while maintaining audiovisual synchronization? Please report changes in F1 scores, MAE, and observed error types to assess the robustness of the main conclusions.