On the Capability Boundaries of Turn-Based GUI Agents in Real-Time GUI Environments: A Human Perception–Action Perspective

GUI agents, also known as computer-using agents (CUAs), have demonstrated strong performance on GUI benchmarks, yet struggle with real-time tasks that humans perform easily. We hypothesize that this gap arises because the turn-based interaction paradigm of current GUI agents is misaligned with the continuous, parallel process by which humans perceive and act when interacting with computers. This misalignment therefore imposes systematic capability boundaries on turn-based GUI agents. To characterize these capability boundaries, we introduce RACE-Bench (Real-time Atomic Capability Evaluation Benchmark), a 69-task real-time GUI benchmark built by extending OSWorld, and organize its tasks by perception and action requirements: A: continuous perception tasks, which require ongoing perception but no time-critical response; B: anticipatory action tasks, which require precisely timed actions selected before execution but no ongoing perception; C: reproducible real-time tasks, which couple perception and action under dynamics that repeat across attempts; and D: stochastic real-time tasks, which couple them under dynamics that vary across attempts. Access to the complete recording up to the current time and within-turn action sequences substantially raise success on A tasks and B tasks, respectively. With both capabilities, agents can also solve some C tasks when retries expose useful experience. D tasks reveal a timing constraint: information needed for a time-sensitive action may become available only shortly before its deadline. If this response window is shorter than the agent’s observation–reasoning–action latency, the agent cannot respond in time. Taken together, these findings show that turn-based agents can compensate for perception or action misalignment in isolation through observation histories or action sequences, but struggle with coupled real-time perception and action unless the dynamics are reproducible and retries allow experience transfer.