acceptodds
Under review as a conference paper at ICLR 2027

Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?

18 views

Abstract

Real-time interaction requires agents to perceive evolving environments and act before opportunities disappear, yet existing evaluations for real-time streaming models focus on conversational quality and streaming perception, leaving their ability to act effectively under strict temporal constraints untested. We first formalize real-time capabilities for agents and introduce Realtime-Bench, a suite of dynamic game and web tasks in which delayed actions directly affect success. We evaluate six frontier LLMs spanning streaming models and standard asynchronous models under three clock protocols at low and high reasoning effort. Wall Clock advances environments with actual inference latency, while Pause Clock removes inference delays as an idealized reference. We further introduce Token Clock, which advances simulated time by the generation tokens consumed before action commitment, testing temporal awareness independently of physical serving speed. Best success rates reach only 32.0% under Wall Clock and 54.0% under Token Clock, compared with 84.0% under Pause Clock, revealing a substantial gap between task competence and timely action. Higher reasoning effort yields mixed effects under time pressure, reflecting a trade-off between decision quality and timely execution. Based on these results, we discuss future directions for improving real-time multimodal models, including adaptive reasoning with temporal awareness and continuous perception and action.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Agents, Agent Evaluation, Vision-Language Models, Real-Time Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JKU3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Realtime-Bench, a benchmark of 30 web tasks and 20 game tasks that tests whether multimodal models can act in time while the environment keeps changing. The environment advances under one of three clocks. Wall Clock advances it by the actual inference latency, Paused Clock pauses it during inference, and the proposed Token Clock advances it according to the number of generated tokens. Across six models, the best success rate is 32% under Wall Clock, 80% under Paused Clock, and 54% under Token Clock. Higher reasoning effort helps every model under Paused Clock, but its effect is mixed under time pressure.

Critical strengths
  1. The large drop from paused to real-time evaluation is a useful finding. On the same tasks with high reasoning effort, the four asynchronous models solve 58% to 80% of the tasks under Paused Clock but only 6% to 32% under Wall Clock. This suggests that strong results on benchmarks that pause the environment may not carry over to real-time use.

  2. The web tasks extend real-time evaluation beyond games, and the benchmark is fully documented. Most real-time evaluations in the related work are game-based, whereas the 30 web tasks involve dynamics such as competing auction bids, expiring price quotes, and seats that sell out. Task specifications, prompts, and the annotation codebook are all in the appendix, and the code is released, so the benchmark is easy to reuse.

Critical weaknesses
  1. Token Clock has a direct precedent, and the paper does not show that it measures temporal awareness. Token Clock is the paper's main methodological contribution, yet the ICLR 2026 paper by Wen et al., Real-Time Reasoning Agents in Evolving Environments, already advances the environment by the number of generated tokens, and the submission does not cite it. The paper also does not report how many tokens each model uses per decision on average, and it does not test whether models shorten their reasoning when tokens cost time. The lower success under Token Clock may therefore simply reflect each model's default output length.

  2. The scores do not separate real-time ability from interface limits and operation errors. The paper does not show that the tasks are solvable through the agent interface, which gives one screenshot per second and allows at most one action per observation. It also does not state which interface the humans in Figure 4 used to reach 100%. Even under Paused Clock, the best success rate on game tasks is only 60%. Many failures therefore occur without any time pressure, so low scores under time pressure cannot all be attributed to weak real-time ability. In addition, the paper does not say how many times each setting was run, and several reported effects of higher reasoning effort amount to only 1 to 4 tasks out of 50 and may reflect random variation.

Questions for authors
  1. How does Token Clock differ from the approach of Wen et al. at ICLR 2026, which also advances the environment by the number of generated tokens? Could you report the average number of tokens each model uses per decision?
  2. What interface did the human baseline in Figure 4 use? Could a human or a scripted agent attempt all tasks through the same agent interface under Paused Clock? Could you also compare the three clocks only on tasks solved under Paused Clock, to isolate the loss caused by time pressure?
1 comment

Thank you for the detailed response and the additional experiments. Two concerns remain.

First, the precedent. I appreciate that the revision now cites Wen et al. and describes Token Clock as adopted. This settles the attribution, but Token Clock is then no longer a methodological contribution of this paper, and the contribution rests on the benchmark and the empirical findings.

Second, what Token Clock measures. The abstract states that Token Clock tests temporal awareness. However, the new experiment shows that at high reasoning effort three of the four models do not shorten their reasoning when tokens cost more time, and at low reasoning effort none of them does. Without such adaptation, the Token Clock score mainly reflects each model's default output length. The shortening experiment measures temporal awareness more directly, because it compares the same model under different token costs. I suggest revising the claim in the abstract accordingly and presenting this experiment as the measure of temporal awareness.

Based on these observations, I maintain my original score.

Reviewer_bMHCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Realtime-Bench, a benchmark for evaluating the interaction capabilities of multimodal agents in continuously changing environments that require time-sensitive actions. The benchmark comprises 30 web-based tasks and 20 game-based tasks and uses three timing protocols: Wall Clock, which accounts for actual inference latency; Paused Clock, which keeps the environment static during model inference; and Token Clock, which advances the environment according to a fixed token-to-time ratio. The authors evaluate six models at different levels of reasoning effort and find that success rates decline substantially when latency is taken into account. They also find that the benefits of increased reasoning effort are constrained by its associated time costs and identify three categories of failure modes through the analysis of specific execution trajectories.

Critical strengths
  1. The research addresses a practically significant problem. Real-world interaction tasks require both correct actions and timely responses and execution. By incorporating time constraints, the benchmark fills a gap in traditional evaluation frameworks, which often overlook response timing.

  2. The three timing protocols provide complementary perspectives. In particular, Token Clock normalizes the token-to-time mapping to neutralize differences in physical service speeds, providing a valuable means of examining the trade-off between decision quality and time expenditure at different levels of reasoning effort.

  3. The paper is well structured and clearly presents its evaluation design and findings. It covers both web-based and game-based tasks, provides a clear analysis of reasoning effort, includes a token-rate ablation study, and presents failure-mode case studies. These elements facilitate understanding of the evaluation design and key findings.

Critical weaknesses
  1. The additional empirical value of Token Clock is not yet fully demonstrated. I acknowledge that §2.2 discusses related work and explains how Token Clock decouples evaluation from actual service speeds. However, prior work has already examined related issues. VideoGameBench [1] explores agent interaction under both real-time and paused-inference conditions, while Gaia2 [2] examines the effects of inference latency on time-constrained tasks and the trade-off between more intensive inference and timely responses. The paper’s primary conclusions remain largely consistent with these prior observations. My concern is that the experiments do not yet sufficiently demonstrate what additional, practically meaningful insights Token Clock provides into model capabilities or the reasons for failure.

  2. The interpretation of cross-model comparisons under Token Clock requires more robust validation. A unified token-to-time mapping can mitigate the effects of physical service latency to some extent, but tokenization schemes and provider-specific definitions of tokens may not be fully aligned. An identical conversion rate therefore does not necessarily assign the same simulated time cost to comparable content generated by different models. I would like the authors to clarify the extent to which the observed performance differences reflect the efficiency of model decision-making and generation, and the extent to which they depend on how tokens are measured.

  3. The analysis of the reported phenomena lacks sufficient depth and evidence of reliability. Although the paper reports non-monotonic trends and identifies three failure categories, several distinct phenomena remain unexplained. In Supplementary Tables 5 and 6, for example, Kimi’s overall gains from increased reasoning effort under Token Clock are driven by web-based tasks, with no improvement on game-based tasks. Similarly, Supplementary Table 7 shows that GPT-Realtime-2 achieves a higher success rate at B = 64 (22.0%) than under Paused Clock, corresponding to B = ∞ (16.0%). The analysis is insufficient to distinguish stable task or strategy characteristics from run-to-run variation. Consequently, the findings provide only limited insight into the factors determining the real-time performance of different models.

References

[1] Alex L. Zhang et al. VideoGameBench: Can Vision-Language Models complete popular video games?. arXiv:2505.18134v1, 2025.

[2] Romain Froger et al. Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments. arXiv:2602.11964v1, 2026.

Questions for authors
  1. What additional information from Token Clock changes the interpretation of a model’s capabilities or the reasons for its failure? Using the existing task-specific results, could the authors provide a concrete example that goes beyond confirming that higher latency reduces success rates?

  2. Which cross-model conclusions are robust to changes in the token-to-time conversion rate, and which depend on particular experimental settings? Could the authors clarify this using the existing token-rate ablation studies? To what extent might the composition of the tokens charged by Token Clock and their length distributions across models affect these conclusions?

  3. Is there evidence—such as from repeated runs or detailed execution traces—that the domain-specific differences and non-monotonic performance patterns in the supplementary tables are stable? For example, which specific tasks and behavioral shifts drive the differences in Kimi’s reasoning gains under Token Clock between web-based and game-based tasks? Do the existing task-level results also support an analysis by latency sensitivity, distinguishing tasks whose performance degrades substantially with even small increases in time cost from tasks that are less sensitive to reasoning latency? I believe such an analysis would demonstrate the diagnostic value of Token Clock more clearly.

Reviewer_zQt5OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Realtime-Bench, a benchmark of 50 game and web tasks for evaluating real-time interaction in multimodal agents. It characterizes this capability through perception, deliberation, and action, and evaluates six models under three clock protocols and two reasoning settings. Token Clock converts generated tokens into simulated time independently of serving speed. The experiments show that task success generally declines when inference delays action execution. Higher reasoning effort improves performance under Paused Clock but has mixed effects under time pressure.

Critical strengths
  1. The paper proposes Realtime-Bench, which combines Wall Clock, Paused Clock, and Token Clock to compare performance under different timing assumptions. Token Clock is a useful addition that preserves the temporal cost of generation without tying action timing to serving speed.

  2. This paper provides a concrete test suite for evaluating LLMs in settings where both action choice and timing matter. Its game and web tasks offer a starting point for studying interaction with changing environments under time constraints.

Critical weaknesses
  1. Evaluating how LLMs perform in real-time interactive settings is an important research direction, but Wordle and Gomoku mainly test decision-making under time limits, which differs from continuously responding to environmental changes. Reporting results mainly by Game/Web makes it difficult to see how different timing mechanisms affect model performance. I suggest breaking down the results by the capabilities each task requires. The benchmark also contains only 50 tasks, all within games and web interactions, limiting its coverage of real-time LLM applications.

  2. The authors distinguish perception, deliberation, and action as three dimensions of real-time capability, but the main evaluation still relies on overall task success rates. Although §6.3 categorizes failures, the paper lacks controlled evaluations of each dimension and metrics such as the fraction of actions that miss their deadlines or response latency to dynamic events. This makes it difficult to identify the models’ specific bottlenecks.

  3. All models are evaluated through external APIs, so Wall Clock results may be affected by network latency and service load. I suggest adding experiments with locally deployed models under fixed hardware and serving configurations to measure actual latency, and comparing their Wall Clock and Token Clock results to assess whether the two timing schemes support consistent conclusions.

Questions for authors

See weaknesses

Comments on the paper

1 comment

We sincerely thank all reviewers for their thoughtful and constructive feedback, which has substantially improved our paper. For the reviewers' and AC's convenience, we summarize the reviews and the additional experiments below.

ReviewerBefore RebuttalAfter RebuttalResponded
zQt512✓
bMHC22✓
JKU322Pending

Summary of Reviews and Our Responses

We are encouraged that the reviewers found real-time interaction an important problem (zQt5, bMHC), found Token Clock a useful complement to Wall and Paused Clock (zQt5, bMHC), considered the large drop from paused to real-time evaluation a useful finding (JKU3), and valued the web tasks beyond games and the fully documented benchmark (zQt5, JKU3). Reviewer bMHC noted that our tokenization and rate analyses help address their Token Clock concerns. The revision also expands evaluation from 6 to 12 frontier models (§6.1, Fig. 1).

ConcernReviewersOur Response
Relation to prior work (Wen et al., VideoGameBench, Gaia2)bMHC, JKU3Cited Wen et al. and now "adopt" Token Clock; clarified our contribution as the first benchmark to evaluate real-time capabilities across diverse scenarios using easily verifiable metrics. (§2.2)
Value of Token Clock beyond "latency hurts"bMHCShowed that the three clocks separate long deliberation from slow serving (R5)
Temporal awarenessJKU3Tested whether models shorten reasoning when tokens cost time; most do not (R6)
Tokenizer and rate sensitivitybMHCCompared tokenizers; separated rate-robust from rate-dependent conclusions (R7)
Per-capability diagnosis and timing metricszQt5, bMHCBroke down results by capability and time sensitivity; reported deadline-miss and latency metrics (R1–R3)
Task scopezQt5Clarified that Wordle/Gomoku isolate RT-Deliberate and that tasks span eight categories (Fig. 3)
API latency in Wall ClockzQt5Evaluated a locally deployed model on fixed hardware (R4)
Interface limitsJKU3Verified solvability and isolated the loss from time pressure (R8)
Human baselineJKU3Clarified the interface and added a 27-participant human study (R9)
Variance and significancebMHC, JKU3Repeated Wall Clock three times; added significance tests and trajectory analyses (R10, R11)

Summary of Additional Experiments

TableQuestion AddressedKey FindingReviewersRevision
R1Which capability limits each model?Streaming models are competitive on RT-Act (GPT-Realtime-2: 22% vs. 14% avg.); asynchronous models lead on RT-Perceive; higher effort often hurts RT-DeliberatezQt5Response
R2How does time sensitivity matter?Streaming models trail on low-sensitivity tasks but beat several asynchronous models on medium-sensitivity oneszQt5, bMHCResponse
R3How often are deadlines missed?48.9% window-overrun rate; only 53.5% of web events answered in timezQt5Response
R4Do results depend on API latency?Local GLM-5.3-Flash: 24.0–29.3% vs. 12.0–12.7% via API; lower latency helps, but success stays below 30%zQt5Response
R5What does Token Clock add?Kimi-K3 and Gemini 3.8 Flash tie at 60% in PL, but Kimi-K3 drops mainly under Token (24%, long deliberation) and Gemini under Wall (28%, slow serving)bMHC, JKU3Fig. 6
R6Do models adapt reasoning to time cost?Only GPT-5.6 Sol at high effort shortens its reasoning (0.81× at B = 32 vs. Paused)JKU3Response
R7Robust to tokenizer and rate B?Median tokenizer gap vs. GPT-5.6 Sol: +11% (Gemini), 0% (Kimi); asynchronous models beat streaming models at every B; finer rankings varybMHCFig. 7
R8Interface or time pressure?47/50 tasks solved under Paused; ≤1.8% unparsable actions; of Paused-solved tasks, 45–57% fail under Token and 73–79% under WallJKU3Response
R9How do humans perform?Humans vs. best agent: 72.0% vs. 31.7% on games, 7.3% vs. 52.2% on webJKU3§6.2, App. C
R10Stable and significant?Best Pass^3 36% over three runs; the effort gain under Paused (p < 0.001) disappears under time pressure (p < 0.001)bMHC, JKU3Fig. 1, App. D.1

On the significance of our findings (bMHC): beyond the known trade-off, our results across tasks and models show that frontier models rarely shorten reasoning when tokens cost time (R6), that streaming models' Wall Clock competitiveness stems from serving speed and does not hold under Token Clock (§6.3, R7), and that the benefit of higher effort disappears under time pressure (R10).

We thank the reviewers again for their valuable feedback and would be happy to provide any further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.