acceptodds
Under review as a conference paper at ICLR 2027

CUAWorld-RT: Can Vision-Language Agents Think and Act in Real Time?

Abstract

AI agents have made substantial progress in domains where they have a non-trivial amount of time to think between actions, such as mathematics, software engineering, and performing tasks on desktop applications. However, it is unclear how such models fare in settings that demand real-time interaction, such as playing a video game or operating a physical device. In these settings, the environment state may evolve as the agent is reasoning, or even as the underlying model is performing a forward pass. To gain insight into the real-time interaction capabilities of vision-language agents, we construct CUAWorld-Realtime (RT). Our key idea is to construct visual minigames that demand a portfolio of capabilities required for interactive real-time settings. We build an agentic pipeline for automatically generating such environments based on a taxonomy of six capabilities required for real-world and real-time interaction. comprises 100 minigame families and 1,000 configurations that vary across task difficulty and the action interface. A key feature of CUAWorld-RT is that it allows for controlled comparisons across different capabilities, such as real-time interaction, temporal reasoning, and exploration of the action interface. Using this, we evaluate frontier and open-weight models. Even GPT-6 Astra solves only 12.5% of real-time tasks at the highest difficulty, while being extremely slow. We also find that models' strong coding capabilities can partially offset their weak real-time capabilities, although their performance remains significantly low. We will publicly release our environment generation framework to spur further research on AI agents in real-time interactive settings.

Primary area:
datasets and benchmarks
Keywords:
real-time agents, benchmark, computer-use-agents, vision language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ee3fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds CUAWorld-RT, a benchmark of 100 small visual minigames. Each game has 10 configurations: 5 difficulty levels, with simplified or full controls. LLM agents write the games and an oracle solver, and a separate audit agent checks them. Other agents label which of four capabilities each configuration needs and whether it needs real-time action. Each configuration can be run live or paused while the model thinks.

The authors test Qwen3.5-9B, Qwen3.8-27B, GPT-5.6 Luna and GPT-6 Astra. Pass rates are low, especially on the 246 real-time configurations (Astra gets 12.5% at difficulty 5). Pausing raises Luna’s real-time pass rate from 8.8% to 32.5%. Letting Astra write scripts raises its rate from 12.5% to 34.4%. Interface rules and simplified controls help Luna. One author gets 78.1% on the hardest real-time set. The authors conclude that agents fail in real time both from missing capabilities and from slow responses.

Critical strengths

None

Critical weaknesses

The evaluation is thin and uneven. for example, Astra is the only model that does reasonably well. It is run only at difficulty 5, with low reasoning effort, while Luna uses xHigh. Every other analysis (pausing, capabilities, rules, difficulty) uses only Luna, which solves 0% of real-time tasks at difficulty 5.

not sure how to compare the time and cost: they depend on how each model is served (Qwen via Tinker, GPT via Codex). The 80 to 145 minutes for Qwen may mostly reflect infrastructure i guess?

Questions for authors

Are the Qwen time numbers dominated by serving latency? Would the results change with matched per-call latency?

Reviewer_5Lf1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CUAWorld-RT, a benchmark for studying visual agents in real-time interactive tasks. It contains 100 minigame families, each with five difficulty levels and two interaction modes. The authors use a creator–auditor pipeline to build the environments and compare agents under live and paused execution, simplified controls, and additional interface information. They also study whether agents can write visual controllers to improve their performance. The results show substantial difficulty on real-time tasks and suggest that coding can help, but some of the comparisons change more than the factor they aim to study.

Critical strengths
  1. The benchmark has a useful design. Varying difficulty, controls, and timing within the same game family can help explain agent failures beyond an overall success rate. The authors also provide detailed construction and auditing procedures, including checks of the action interface and oracle solutions.

  2. The coding results are interesting. On the hardest real-time subset, Astra improves from 12.5% to 34.4% when allowed to write visual controllers. The examples of template matching and motion estimation, including unsuccessful attempts, help explain how these controllers work and where they still fail.

Critical weaknesses
  1. My main concern is that the live/paused comparison does not isolate the effect of inference delay. Appendix E states that live observations provide a single current frame, while paused observations can provide a sequence of one to eight frames. Figures 9–10 also show different behavior for held inputs: a down–wait–up sequence advances the live game but not the paused game. Therefore, the improvement from 8.8% to 32.5% may partly come from additional temporal information or different control behavior. Since diagnosing the source of real-time failures is central to the paper, I think this comparison needs a matched protocol.
Questions for authors
  1. Can the live/paused experiment use the same observation frames and action-hold behavior, changing only whether the game advances during model inference? Does the large performance gap remain under this control?
  2. Have independent human players tested a representative sample using the exact screenshot and action interface available to the agents? How often do they encounter insufficient visual information or disagree with the assigned capability labels?
Reviewer_25y3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CUAWorld-RT, a benchmark evaluating models that operate mouse and keyboard controls from screenshots of generated minigames. A model may choose a sensible action but take so long to decide that the environment has changed before it acts. The minigames vary requirements such as following moving objects, interpreting spatial relationships, and discovering unfamiliar controls. CUAWorld-RT tries to separate these possible reasons for failure using configurable minigames, rather than treating a low game score as a diagnosis by itself. The suite varies game difficulty, control interfaces, and whether time continues during model reasoning. A configuration is a particular game, difficulty, and interface. In live mode, the game keeps running while the model decides. Paused mode advances through an observation window, returns its frames, and freezes until an action executes. Researchers can then compare live and paused versions, supply instructions, or simplify controls. Examples include navigating with fading visual information and interacting with moving keys. The intended logic is to change a suspected obstacle and observe whether performance improves; identifying that obstacle as the cause requires knowing what else the intervention changed.

The contribution is a minigame collection with adjustable controls, task clocks, and labels describing the abilities each game is intended to require. It enables studies of how agent performance changes when instructions, controls, or time pressure change. Generation pipelines and pausing are established ideas; their integration into this collection is the resource contribution. Automated creator agents implement games and solution programs; auditor agents check execution and recorded evidence; other model-based review agents assess difficulty and task demands. The paper uses the resulting suite to seek explanations for agent failures, although successful interventions alone do not isolate their causes.

Frontier and open-weight model evaluations measure success, attempt duration, and inference cost. Solution programs can read internal game state but act through the evaluation controls, demonstrating that objectives are reachable. Audits and initial manual review support construction quality; an author's success on much of a difficult real-time subset provides additional evidence of usability through the visible interface. This does not establish screenshot-only solvability across the entire suite. Pausing improves Luna on both real-time and non-real-time task groups. Paused observations may include a motion sequence, whereas a live observation returns a current screenshot. The paper reports ranges of observation settings but does not identify the settings used by each minigame in this comparison, so the gain cannot yet be attributed solely to eliminating inference delay. Rulebooks describe controls and fixed rules without providing instance-specific solutions; adding them improves Luna on tasks requiring interface exploration. Coding lets the Astra model write programs that process screenshots and issue the same mouse and keyboard actions through an API, without internal-state access, and improves on its ordinary tool-by-tool interaction. Simplified interfaces can replace difficult controls with easier operations, such as typing instead of clicking moving keys. With simplified controls and paused execution, Luna has lower success on tasks labeled as requiring temporal reasoning or 3D visual understanding than on tasks without the respective label; these are associations across different task groups. The authors acknowledge overlapping labels and show that the temporal-reasoning gap largely disappears among non-real-time tasks. Other demands still differ between groups, so these comparisons do not isolate a model's individual capability deficits. Overall, the evidence supports useful tasks and practical ways to improve agent interaction more firmly than it establishes a clean diagnosis of the abilities causing failure.

Critical strengths
  1. The suite supports reusable experiments on visual interaction: researchers can hold a game fixed while varying controls, difficulty, or whether it advances during model reasoning. Executable solutions and audits check functioning tasks, while the human case study supplies limited evidence of success through the visible interface. This makes the collection a useful experimental resource, even without treating its labels as calibrated measurements of isolated abilities.

  2. The interventions identify practical improvements. Supplying interface rules helps Luna without supplying answers; allowing Astra to generate screenshot-processing action programs helps on the hardest real-time tasks compared with its ordinary interaction loop. The coding condition changes how actions are generated and scheduled, so it is evidence for a useful agent setup rather than a pure test of one capability. These results support studying agent designs with this testbed.

Critical weaknesses

The paper's attempt to diagnose failure causes is central, but the evidence supports narrower conclusions. For latency, the claim is that slow model responses help explain poor real-time success. Appendix E specifies a current screenshot for a live observation and potentially several chronological frames for a paused observation. A sequence may provide motion information beyond a single frame. Live agents can request successive screenshots, but the paper does not establish matched observation timing and packaging for the configurations in Figure 5. I therefore cannot determine whether the reported benefit isolates response delay. This is an unresolved protocol question, not proof that extra frames caused the gain.

The authors acknowledge overlapping capability labels and partly address them by separating real-time and non-real-time configurations. Nevertheless, lower success across the remaining task groups does not identify an isolated ability deficit. These limits weaken the diagnostic interpretation. I nevertheless lean toward acceptance for the configurable testbed and demonstrated ways to improve interaction; those contributions remain useful without assigning each failure to a specific cause.

Questions for authors

Could you add a controlled experiment separating response latency from observation and control differences? This would help establish whether slow responses cause the failures, which the current live/paused comparison cannot isolate because it also changes how observations and actions are delivered.

Could you validate a capability label through matched task variants—for example, by explicitly providing motion information in a task that otherwise requires inferring it? This would help establish whether the labeled capability explains the difficulty; the current comparisons across different tasks leave other task differences as possible explanations, despite the reported stratification.

Could you demonstrate an experiment enabled by CUAWorld-RT’s additional controls that the closest existing benchmark cannot readily support? This would clarify why researchers need this resource; the related-work discussion describes differences but does not clearly demonstrate what additional scientific insight they enable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.