acceptodds
Under review as a conference paper at ICLR 2027

RTWorld: Benchmarking Real-Time Game Playing Agent for Open-Ended Tasks in Digital Environment

Abstract

Generalist visual agents must perceive, plan, and act while the digital world continues to change, yet most evaluations implicitly let the world wait for model inference. This convention obscures whether an agent can turn a correct decision into a timely action. We introduce RTWorld, a benchmark for real-time game-playing agents built from 31 open-ended browser games with pixel-only observations, mixed keyboard and pointer control, and verifiable native outcomes. A fail-closed runtime preserves the live world clock, timestamps the full observation–inference–action path, rejects invalid or stale plans, binds each episode to one model replica, and records replayable hash-verified evidence. We evaluate 13 vision-language systems using common seeds and complement the main benchmark with controlled-delay, pause/live, delay-shape, and temporal-context interventions. Our results show that static model quality, response availability, and closed-loop action freshness are distinct factors: model rankings can change when inference no longer pauses the environment, and faster responses do not by themselves guarantee useful control. RTWorld provides a reproducible foundation for studying agents that must reason and act before the opportunity has passed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Real-Time Agents, Game-Playing Agents, Multimodal Agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YUmzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

A benchmark is introduced built from 31 open-ended browser games. The paper is difficult to follow. Many sentences consist of a long list of somewhat related topics that are not explained in the paper. The paper uses much jargon, is difficult to understand by outsiders. The paper contributed a benchamrk. No research questions are posted, no answers are given, no conclusion is present. This is not written in the style of a scientific paper. Please rewrite or submit to a suitable venue.

There is no supplementary material. No AI use statement.

reject

Critical strengths
  • a benchmark is introduced of substantive size for real time web interaction
Critical weaknesses
  • paper is difficult to follow
  • paper reports on the benchmark, but has no research questions, conclusions, or research insights.
  • no supplementary material, no AI use statement.
Questions for authors

ICLR 2027 RTWORLD: BENCHMARKING REAL-TIME GAME

What is a fail-closed runtime? Abstract states that 13 VLM are evaluated with common seeds. How can it be relevant that seed values of a random number generator are shared between VLMs? What is meant by "distinct factors", does that mean relevant, or determining? What does it mean that faster responses do not by themselves guarantee use-ful control? What does it mean that the opportunity has passed, which opportunity: RTWorld provides a reproducible foundation for studying agents that must reason and act before the opportunity has passed.

Abstract must be rewritten.

Real world: environment may move on while agent is thinking. Writing style is difficult: many sentences with long item lists, that are not explained further on: "Without replica identity, immutable traces, synchronized video, and explicit exclusions, apparent gains can arise from hidden pauses, stale actions, mid-episode failover, or incomplete evidence."

What is a source-audited browser game? Who is the auditor? What is DOM state?

"The world clock advances through capture, inference, ad- mission, and dispatch. Schema, interruption-epoch, and source-age checks prevent invalid or stale plans from controlling the game. Each episode is bound to one model replica and yields hash-linked inputs, outputs, traces, results, and video." Please rewrite, and move out of introduction, to experimental setup.

What is freshness? What does this sentence mean: "Latency and freshness thus become observable properties of the evaluated system rather than experimental noise." How can freshness be experimental noise? What is experimental noise?

The list of contributions should be rewritten using full sentences. There are no scientific insights listed as contribution, only a benchmark set.

What does this sentence mean: "Dustline is implemented but not evaluated here."

Related work is unreadable. Please rewrite using english sentences. What are the conclusions, how is it related to your work? Please have a look at other relevant scientific papers, and produce a paper in a proper scientific format, where research questions are posed and answered, or follow a course in academic writing. Note that ICLR is one of the premier AI conferences, and standards are high, for readability and scientific contributions.

Section 7, results, does not have conclusions. Please provide a clear problem statement, and list contributions to the scientific community.

The final sentence of the conclusion is a riddle: The evidence is deployment-conditioned, with no supported rank reversal or burst-ordering effect; broader cross-family replication remains essential.

I do not think that this paper belongs at ICLR in its current form.

reject

Reviewer_yMnZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RTWorld, a benchmark for evaluating the ability of vision-language models (VLMs) to interact with games in continuously evolving environments that do not pause during model inference. The benchmark comprises 31 browser-based games and evaluates 13 deployed systems. The authors analyze the effects of temporal factors on performance through experimental variations, including additional response latency and pausing the environment during inference. They also propose three metrics—action freshness, response admission rate, and control coverage—to characterize the agent’s interaction process in greater detail.

Critical strengths
  1. The research problem is practically relevant. In real-time interaction, selecting the correct action is insufficient unless the action is also executed in a timely manner. Examining the interplay between inference latency and environmental dynamics helps explain agent performance in realistic scenarios.

  2. The evaluation framework provides detailed temporal logs and diagnostic metrics. Beyond final scores, these measurements support an assessment of whether actions arrive on time, whether they are successfully executed, and the extent to which the agent effectively controls the environment.

Critical weaknesses
  1. The specific contributions relative to prior work are insufficiently clear. VideoGameBench [1] has already addressed stale actions caused by inference latency, including a Lite mode that pauses the game during inference. GameWorld [2] provides a browser-based game evaluation framework, and its GameWorld-RT variant specifically examines agent performance in continuously evolving environments. Although the manuscript cites these works, it does not sufficiently explain the unique evaluation capabilities offered by the proposed protocols and task designs. The paper describes several functional differences, but why these differences constitute a significant research contribution remains unclear.

  2. The value of the new diagnostic metrics is not fully demonstrated. Although action freshness, response admission rate, and control coverage have practical value, the paper does not adequately demonstrate how they identify failure causes that are difficult to distinguish using task scores or latency metrics alone. It also does not sufficiently show how these metrics could inform the selection of specific inference, scheduling, or action-execution strategies. In my view, the current analysis does not clearly establish the unique insights provided by these measurement capabilities or how such insights could benefit future research or system design.

  3. The AI use statement could not be located, and Figure 3 has a readability issue. I could not locate an AI use statement in the submitted manuscript, although the ICLR 2027 Author Guidelines [3] explicitly require one. Please indicate its location or provide the statement. In addition, the blue arrow connecting “Async VLM” to the “Freshness gate” in Figure 3 (page 5) obscures part of the text describing the “assigned holdback” mechanism, hindering understanding of the execution flow. I suggest adjusting the text and arrow layout.

References

[1] Alex L. Zhang et al. VideoGameBench: Can Vision-Language Models complete popular video games?. arXiv:2505.18134, 2025.

[2] Mingyu Ouyang et al. GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents. arXiv:2604.07429, 2026.

[3] ICLR. ICLR 2027 Author Guidelines, “Required: AI use statement.” Accessed October 17, 2026.

Questions for authors
  1. What is the most critical new evaluation capability that RTWorld introduces relative to VideoGameBench and GameWorld? Which reported results best demonstrate the unique value of this capability?

  2. Could the authors provide a concrete diagnostic example showing how the new temporal metrics distinguish failure causes that are difficult to identify from task scores or inference latency alone?

  3. How do the game and task designs specifically embody an “open-ended” nature? What new dimensions of capability or evaluation challenges does this work introduce relative to existing game-based benchmarks?

Reviewer_rFJiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

RTWorld tests vision-language models on 31 browser games that keep running while the model thinks. The runtime timestamps capture, inference and dispatch, rejects stale or malformed plans, ties each episode to one model replica, and hashes the evidence for every episode. The main evaluation covers 13 open-weight systems on ten shared seeds, for 4,030 episodes. On five games and three Qwen models, the authors also add fixed response delays, paused-versus-live pairs, different delay patterns and changes to the temporal context. The paper argues that static model quality, response availability and action freshness are different factors in real-time play.

Critical strengths
  1. The setting is worth studying, since the environment keeps changing while the model infers. The paper reports task quality, response admission and fresh active-control coverage separately, so a fast reply is not counted as useful control by default. Shared seeds, assigned delays and paired paused and live runs are a reasonable way to study whether an agent acts before an opportunity is gone.

  2. The reporting is thorough and includes negative results. The main panel has 4,030 episodes with seed-block intervals, checks on how the score is defined, leave-one-game-out rankings, and latency summaries both pooled over events and balanced over episodes. Normalization gates that failed and effects that were not supported are reported openly.

Critical weaknesses
  1. Many evaluated systems score close to a random policy, which makes it unclear what the benchmark separates. The uniform-random baseline scores 0.197 [0.181, 0.217]. Four systems have lower point estimates (Qwen2.5-VL-3B, 7B and 32B, and Gemma-4-31B), and four more fall inside the random baseline's interval (0.202 to 0.214). Overlapping intervals are not a test, and the random baseline was run on independent seeds, so the paper should report model-minus-random differences with intervals. The random policy also scores above zero in 69.0% of episodes, equivalent to about 21.4 positive-score games per 31-game round, compared with the highest model coverage of 21.3 games for Qwen3.8-27B. There is no human baseline and no proprietary model to anchor the top of the range, and the score thresholds are set by the designers. Gemma-4-31B has a median model latency near 50 seconds and dispatches 8 actions across 310 episodes. Since the benchmark measures deployed systems, this can be a legitimate result, but the serving setup needs explaining, as does how it fits the rule that formal scoring starts after one valid admitted action.

  2. Some headline claims go further than the statistics, and the delay experiments are hard to separate from the gate. The abstract says rankings can change when inference no longer pauses the environment, but the paper finds no rank reversal supported by simultaneous bands, and the Qwen3.8 pause gap has an exact p of 0.0547 with a paired-t interval that includes zero. The body reports these results honestly, and the abstract should match it. In the delay study, the 1,400 ms plan-age gate excludes the 1,500 and 2,000 ms conditions outright, and at 1,000 ms admission falls to 92.5%, 10.9% and 0.2% for the three models, so the curves mostly describe the deployed protocol as a whole. The authors acknowledge this issue and restrict the primary AUC to 0–1,000 ms, but even within that range, the results still combine the effects of delayed control and admission-gate rejection. The abstract and introduction are also hard to follow. They pile up runtime terms and qualifications and never state plainly what the experiments found.

  3. The presentation makes the paper unnecessarily difficult to follow. The abstract and introduction pack runtime terminology and qualifications into dense sentences, making the main empirical findings difficult to identify. Figure readability is also poor: Figure 5 uses long, rotated configuration labels in very small text, while the axis, legend, and matrix labels in Figure 6 are substantially smaller than the surrounding body text. These choices make it difficult to compare experimental settings and interpret the figures at normal page size. The paper needs a clearer statement of its main findings, simpler explanations of the runtime and metrics, and larger, more legible figure labels.

Questions for authors
  1. How should readers interpret results where eight of thirteen systems are below or near a uniform-random policy? Can you report model-minus-random differences with intervals, add a human baseline and at least one stronger system such as a proprietary model, and explain the Gemma-4-31B serving configuration?

  2. What is the strongest finding the abstract is meant to convey, and which number supports it? Please align the ranking-change statement with the unresolved comparisons, or mark it as exploratory.

  3. Can the existing traces break each delay condition into gate rejections, admitted actions and outcomes, and show what remains in the range the gate does not exclude? Which maximum plan age does the main benchmark use? The runtime overview shows a 1,600 ms check, while the delay experiments use 1,400 ms. Finally, how should fresh-control coverage be related to score across models, given that the second-ranked system has very low fresh coverage?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.