- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Realtime-Bench, a benchmark of 30 web tasks and 20 game tasks that tests whether multimodal models can act in time while the environment keeps changing. The environment advances under one of three clocks. Wall Clock advances it by the actual inference latency, Paused Clock pauses it during inference, and the proposed Token Clock advances it according to the number of generated tokens. Across six models, the best success rate is 32% under Wall Clock, 80% under Paused Clock, and 54% under Token Clock. Higher reasoning effort helps every model under Paused Clock, but its effect is mixed under time pressure.
Critical strengths
-
The large drop from paused to real-time evaluation is a useful finding. On the same tasks with high reasoning effort, the four asynchronous models solve 58% to 80% of the tasks under Paused Clock but only 6% to 32% under Wall Clock. This suggests that strong results on benchmarks that pause the environment may not carry over to real-time use.
-
The web tasks extend real-time evaluation beyond games, and the benchmark is fully documented. Most real-time evaluations in the related work are game-based, whereas the 30 web tasks involve dynamics such as competing auction bids, expiring price quotes, and seats that sell out. Task specifications, prompts, and the annotation codebook are all in the appendix, and the code is released, so the benchmark is easy to reuse.
Critical weaknesses
-
Token Clock has a direct precedent, and the paper does not show that it measures temporal awareness. Token Clock is the paper's main methodological contribution, yet the ICLR 2026 paper by Wen et al., Real-Time Reasoning Agents in Evolving Environments, already advances the environment by the number of generated tokens, and the submission does not cite it. The paper also does not report how many tokens each model uses per decision on average, and it does not test whether models shorten their reasoning when tokens cost time. The lower success under Token Clock may therefore simply reflect each model's default output length.
-
The scores do not separate real-time ability from interface limits and operation errors. The paper does not show that the tasks are solvable through the agent interface, which gives one screenshot per second and allows at most one action per observation. It also does not state which interface the humans in Figure 4 used to reach 100%. Even under Paused Clock, the best success rate on game tasks is only 60%. Many failures therefore occur without any time pressure, so low scores under time pressure cannot all be attributed to weak real-time ability. In addition, the paper does not say how many times each setting was run, and several reported effects of higher reasoning effort amount to only 1 to 4 tasks out of 50 and may reflect random variation.
Questions for authors
- How does Token Clock differ from the approach of Wen et al. at ICLR 2026, which also advances the environment by the number of generated tokens? Could you report the average number of tokens each model uses per decision?
- What interface did the human baseline in Figure 4 use? Could a human or a scripted agent attempt all tasks through the same agent interface under Paused Clock? Could you also compare the three clocks only on tasks solved under Paused Clock, to isolate the loss caused by time pressure?
Thank you for the detailed response and the additional experiments. Two concerns remain.
First, the precedent. I appreciate that the revision now cites Wen et al. and describes Token Clock as adopted. This settles the attribution, but Token Clock is then no longer a methodological contribution of this paper, and the contribution rests on the benchmark and the empirical findings.
Second, what Token Clock measures. The abstract states that Token Clock tests temporal awareness. However, the new experiment shows that at high reasoning effort three of the four models do not shorten their reasoning when tokens cost more time, and at low reasoning effort none of them does. Without such adaptation, the Token Clock score mainly reflects each model's default output length. The shortening experiment measures temporal awareness more directly, because it compares the same model under different token costs. I suggest revising the claim in the abstract accordingly and presenting this experiment as the measure of temporal awareness.
Based on these observations, I maintain my original score.