This paper presents CUAWorld-RT, a benchmark evaluating models that operate mouse and keyboard controls from screenshots of generated minigames. A model may choose a sensible action but take so long to decide that the environment has changed before it acts. The minigames vary requirements such as following moving objects, interpreting spatial relationships, and discovering unfamiliar controls. CUAWorld-RT tries to separate these possible reasons for failure using configurable minigames, rather than treating a low game score as a diagnosis by itself. The suite varies game difficulty, control interfaces, and whether time continues during model reasoning. A configuration is a particular game, difficulty, and interface. In live mode, the game keeps running while the model decides. Paused mode advances through an observation window, returns its frames, and freezes until an action executes. Researchers can then compare live and paused versions, supply instructions, or simplify controls. Examples include navigating with fading visual information and interacting with moving keys. The intended logic is to change a suspected obstacle and observe whether performance improves; identifying that obstacle as the cause requires knowing what else the intervention changed.
The contribution is a minigame collection with adjustable controls, task clocks, and labels describing the abilities each game is intended to require. It enables studies of how agent performance changes when instructions, controls, or time pressure change. Generation pipelines and pausing are established ideas; their integration into this collection is the resource contribution. Automated creator agents implement games and solution programs; auditor agents check execution and recorded evidence; other model-based review agents assess difficulty and task demands. The paper uses the resulting suite to seek explanations for agent failures, although successful interventions alone do not isolate their causes.
Frontier and open-weight model evaluations measure success, attempt duration, and inference cost. Solution programs can read internal game state but act through the evaluation controls, demonstrating that objectives are reachable. Audits and initial manual review support construction quality; an author's success on much of a difficult real-time subset provides additional evidence of usability through the visible interface. This does not establish screenshot-only solvability across the entire suite. Pausing improves Luna on both real-time and non-real-time task groups. Paused observations may include a motion sequence, whereas a live observation returns a current screenshot. The paper reports ranges of observation settings but does not identify the settings used by each minigame in this comparison, so the gain cannot yet be attributed solely to eliminating inference delay. Rulebooks describe controls and fixed rules without providing instance-specific solutions; adding them improves Luna on tasks requiring interface exploration. Coding lets the Astra model write programs that process screenshots and issue the same mouse and keyboard actions through an API, without internal-state access, and improves on its ordinary tool-by-tool interaction. Simplified interfaces can replace difficult controls with easier operations, such as typing instead of clicking moving keys. With simplified controls and paused execution, Luna has lower success on tasks labeled as requiring temporal reasoning or 3D visual understanding than on tasks without the respective label; these are associations across different task groups. The authors acknowledge overlapping labels and show that the temporal-reasoning gap largely disappears among non-real-time tasks. Other demands still differ between groups, so these comparisons do not isolate a model's individual capability deficits. Overall, the evidence supports useful tasks and practical ways to improve agent interaction more firmly than it establishes a clean diagnosis of the abilities causing failure.