acceptodds
Withdrawn from ICLR 2027

TimeActBench: Benchmarking Time-Aware Actions in Real-Time Interactive Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Temporal precision is fundamental to human-level intelligence, allowing actions to be coordinated with events as they unfold. Recent real-time interactive models enable continuous perception and timely responses during ongoing interactions, opening the way to evaluating this capability beyond traditional turn-based settings. However, existing temporal benchmarks primarily emphasize retrospective temporal understanding or specialized forms of timing-sensitive interaction, leaving the ability to determine whether and when to execute task-specific actions under causal streaming observations insufficiently characterized. To address this gap, we introduce TimeActBench, a benchmark for systematically evaluating time-aware actions in real-time interactive models. TimeActBench comprises seven task families and 196 evaluation instances spanning audio, video, and Audio-Visual settings, and organizes time-aware action into two complementary dimensions: Perception and Detection (PD), which evaluates whether models can detect task-relevant events and emit the prescribed detection action when those events become observable, and Response and Interaction (RI), which evaluates whether models can execute appropriate actions according to temporal constraints imposed by an evolving interaction. Our main comparison evaluates 15 proprietary and open-source real-time interactive models, and we conduct a human study under the same evaluation protocol. Our results reveal substantial limitations in current models, with a pronounced overall gap between model and human performance across action correctness and temporal precision. These findings establish time-aware action as a challenging and underexplored capability of real-time interactive models. We expect TimeActBench to provide a systematic testbed for studying this capability and to advance research on temporal intelligence in real-time interactive models, and it will be fully open-sourced.

Primary area:
datasets and benchmarks
Keywords:
Artificial Intelligence, Temporal Intelligence, Real-time Interactive Model, Temporal Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hwxwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TimeActBench, a benchmark for evaluating whether real-time interactive models can determine, under causal, streaming observations, whether to act and when to execute an action. The benchmark comprises 196 instances across seven task categories spanning audio, video, and audiovisual modalities. It evaluates 15 models alongside human participants on both action correctness and temporal precision. The study reports a significant performance gap between models and humans and notes that low timing error on matched actions can coexist with frequent erroneous triggers.

Critical strengths
  1. The research problem is highly relevant to real-time interactive systems. By shifting the focus from retrospective temporal understanding to action selection and timing as events unfold, the work addresses practical requirements of these systems.

  2. The task design covers diverse modalities and temporal constraints. The evaluation considers both action correctness and timing accuracy. Human baselines help characterize task difficulty, while the supplementary code enables verification of the evaluation implementation.

Critical weaknesses
  1. Differences in observation delivery may confound the attribution of results to model capabilities. In the supplementary code, the Qwen multimodal adapter waits for a full response cycle to complete after committing the input and stops sending subsequent audiovisual input during this period. The evaluation clock continues to advance, after which the system catches up by transmitting the backlog. In contrast, Gemini's media-streaming pipeline does not include this waiting mechanism. Different models may therefore receive observations at different times. Delayed responses, missed detections, or incorrect actions could arise, at least in part, from delayed observations or the deferred delivery of past states, rather than from a model's inability to make correct decisions given sufficient information. The authors should clarify how the supplementary implementation relates to the experiments reported in the paper and quantify the extent to which these differences in observation delivery affect the reported results.

  2. The backlog issue is compounded by uncertainty about the temporal information available to models. Although the video-streaming functions for both Gemini and Qwen accept raw frame timestamps as parameters, the actual requests transmit only JPEG content without explicitly including the associated timestamps. Timestamps are not mandatory inputs for all real-time tasks. However, when input is blocked and subsequently delivered through a catch-up mechanism, the original video timestamp differs from the time at which the image arrives. The authors must clarify how the model identifies or reconstructs the relationship between these two time references. Otherwise, metrics computed against the original video timeline may attribute errors caused by stale observations to deficiencies in the model's temporal reasoning.

  3. The experiments do not sufficiently rule out protocol-related influences on the main conclusions. Appendix I acknowledges that the reported timing reflects the entire deployed system. However, these protocol-related factors may affect not only Mean Absolute Error (MAE) but also action accuracy by altering the observations available to the model. Round-Trip Time (RTT) measurements cannot disentangle the contributions of client-side waiting, input backlogs, and submission mechanisms. The supplementary multimodal configurations also specify 1 FPS video input and a one-second commit interval, making it essential to verify whether task-relevant cues arrive before a response is required. The paper lacks analyses of actual input latency, the observability of target events, and sensitivity to scheduling strategies. It therefore remains difficult to determine the proportion of reported failures attributable to the model itself versus the evaluation framework.

Questions for authors
  1. Did all experiments use the waiting, subsequent catch-up delivery, 1 FPS sampling, and one-second submission settings described in the supplementary code? Please provide representative execution traces showing the original event timestamp, the actual transmission time, and the time at which an action was received, together with input-latency measurements and the frequency with which target cues failed to arrive on time.

  2. What temporal information is actually available to the model? When image requests do not explicitly include the original frame timestamp, how are backlogged observations handled, and how is the relationship between the temporal cues available to the model and the evaluation timeline made explicit?

  3. How does the current scheduling strategy compare with an alternative supported by the same model's API? On a representative set of tasks, could you compare the current strategy with an alternative such as skipping stale observations while maintaining audiovisual synchronization? Please report changes in F1 scores, MAE, and observed error types to assess the robustness of the main conclusions.

Reviewer_QdytOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces TimeActBench, a benchmark for evaluating time-aware actions in real-time audio, video, and audiovisual streams. It includes 196 instances across seven task families, covering both Perception and Detection (PD) and Response and Interaction (RI). The benchmark evaluates 15 models and human participants, revealing a substantial gap between model and human performance in action correctness and timing precision.

Critical strengths
  1. The benchmark uses controlled inputs with reference timestamps and includes human baselines to validate task feasibility. It evaluates both action correctness and timing error, showing that low timing error can still come with frequent false alarms. The modality-specific comparisons and documented latency controls also improve evaluation fairness.

  2. This benchmark test has taken the evaluation paradigm from the traditional "retrospective temporal understanding" to the cross-modal "causal streaming evaluation". It provides a very timely and pioneering testing perspective for evaluating the real-time temporal perception ability of real-time interaction models in dynamic environments.

Critical weaknesses
  1. The evaluation metrics employed in the paper exhibit several deficiencies. Section 3.4 describes label-based one-to-one matching, but Appendix B.5 mentions task-specific temporal rules without defining them. In Table 9, MiniCPM-o 4.5 predicts "hit" at 10.3463s, before the reference at 10.752917s, and it is marked as a false alarm. For spoken responses, it is unclear whether timestamps refer to speech onset, audio arrival, or recognition of the target token. Speech stopping and resumption also lack precise definitions. In addition, frame rates, audio chunk sizes, and buffering settings should be reported for each model interface, as these affect timing measurements and cross-model comparisons. Section 3.4 calculates mean absolute error (MAE) exclusively over correctly matched actions, ignoring false positives entirely. The main comparison in Table 2 unfairly conflates physical network latency with model temporal intelligence by directly comparing cloud-based APIs with locally deployed models.

  2. While the paper provides a initial, protocol-aligned human reference baseline, caution is warranted when leveraging it for model rankings, generalized conclusions, or strong claims of reaching "near-human performance." To support such assertions, a larger sample size, a more diverse participant cohort, and a rigorous uncertainty analysis are essential.

Questions for authors
  1. What are the exact action-matching rules, especially for early, late, and duplicate predictions? Why is MiniCPM-o 4.5's prediction in Table 9 counted as a false alarm?

  2. How are timestamps defined and recorded for spoken responses, control signals, and speech stopping or resumption?

  3. Could you provide the frame rates, audio chunk sizes, and buffering settings for each model interface, including proprietary APIs?

  4. How are the "All Modalities" metrics calculated? Are they based on the same runs as the modality-specific results? Could you provide the reference and matched-action counts and explain Gemini 2.5 Flash Live's overall MAE of 1,621 ms?

  5. If the goal is to enable models to produce outputs at precise times, wouldn't a simple clock tool or a timer be sufficient? I would particularly like the authors to clarify what TimeActBench evaluates beyond such basic timing mechanisms, and why these simpler approaches would not be sufficient.

  6. Since MAE is calculated exclusively on correctly matched actions and ignores false positives, how does the benchmark penalize models that achieve low timing errors through high-frequency guessing (e.g., Qwen 3.5 Omni Plus Realtime's 38 false alarms in Table 10)? Could you provide an updated leaderboard using a unified temporally-windowed metric (such as Temporal mAP or Windowed F1) to prevent this degenerate strategy?

  7. Why does the main comparison in Table 2 retain end-to-end transport latency without normalizing RTT, given that cloud API delays (up to 185.31 ms) account for over half of the human baseline MAE? Could you provide a latency-adjusted performance table (e.g., by subtracting half-RTT from action timestamps) to isolate the models' true cognitive reaction times from network I/O bottlenecks?

Reviewer_dXe6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TimeActBench, a benchmark for evaluating whether and when real-time interactive models execute task-specific actions. It contains 196 instances across seven task families, covering audio, video, and audio-visual settings. The benchmark evaluates both action correctness and execution timing. Experiments with 15 real-time models and human participants reveal substantial performance gaps, particularly in reliable event triggering and temporal precision.

Critical strengths
  1. The paper identifies a meaningful gap between temporal understanding and acting at the appropriate time.
  2. Evaluating both action correctness and execution timing is relevant to real‑time interactive systems.
  3. The benchmark covers multiple modalities and different forms of event‑triggered interaction.
  4. The authors evaluate a broad set of proprietary and open‑source models, together with human participants.
  5. The analysis reveals useful failure patterns, including false activations, missed actions, and increased latency under stronger reasoning settings.
Critical weaknesses
  1. The benchmark is relatively small. Only 196 instances are distributed across seven task families and three modalities. This raises concerns about the reliability and generality of model rankings.
  2. The evaluation relies heavily on synthetic data. Although this allows precise temporal annotation, it remains unclear whether the observed failures generalize to natural conversations and real‑world video streams.
  3. The distinction from existing benchmarks needs stronger empirical support. OVO‑Bench, StreamReady, Game‑Time, and OmniInteract already evaluate aspects of streaming response and timing. The contribution is primarily the unified evaluation of task‑specific actions.
  4. The scoring protocol raises concerns about temporal matching. Appendix F.5, Table 10, counts a response at 10.2498 s as a false alarm for an event at 10.2500 s, while a response almost one second later is counted as correct.
  5. Model comparisons may be affected by deployment conditions. The reported timing includes API scheduling and network latency. Although RTT is measured, these factors are not separated from model inference time.
  6. The statistical reliability of model rankings is unclear. The human study includes repeated trials, but comparable repeated‑run variability is not reported for the evaluated models.
Questions for authors
  1. The benchmark contains 196 instances, many generated under controlled temporal conditions. How representative are these instances of natural real‑time interactions? Can the authors provide evidence that the main findings hold under more varied or less controlled inputs?
  2. Appendix F.5, Table 10, marks a response at 10.2498 s as a false alarm for an event at 10.2500 s, while a response at 11.2160 s is considered correct. Could the authors clarify the temporal matching rule and show how sensitive F1 and model rankings are to reasonable timing tolerances?
  3. The reported timing includes network and service‑side latency. Although Appendix G provides RTT measurements, how stable are the rankings across repeated evaluations or different deployment conditions? Could the authors provide confidence intervals or repeated‑run results?
  4. Some models produce additional speech or outputs that do not follow the prescribed action format. What proportion of failures comes from incorrect temporal decisions versus output‑protocol violations? Could the authors provide a breakdown to clarify what capability the benchmark is measuring?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.