acceptodds
Under review as a conference paper at ICLR 2027

On the Capability Boundaries of Turn-Based GUI Agents in Real-Time GUI Environments: A Human Perception–Action Perspective

Abstract

GUI agents, also known as computer-using agents (CUAs), have demonstrated strong performance on GUI benchmarks, yet struggle with real-time tasks that humans perform easily. We hypothesize that this gap arises because the turn-based interaction paradigm of current GUI agents is misaligned with the continuous, parallel process by which humans perceive and act when interacting with computers. This misalignment therefore imposes systematic capability boundaries on turn-based GUI agents. To characterize these capability boundaries, we introduce RACE-Bench (Real-time Atomic Capability Evaluation Benchmark), a 69-task real-time GUI benchmark built by extending OSWorld, and organize its tasks by perception and action requirements: A: continuous perception tasks, which require ongoing perception but no time-critical response; B: anticipatory action tasks, which require precisely timed actions selected before execution but no ongoing perception; C: reproducible real-time tasks, which couple perception and action under dynamics that repeat across attempts; and D: stochastic real-time tasks, which couple them under dynamics that vary across attempts. Access to the complete recording up to the current time and within-turn action sequences substantially raise success on A tasks and B tasks, respectively. With both capabilities, agents can also solve some C tasks when retries expose useful experience. D tasks reveal a timing constraint: information needed for a time-sensitive action may become available only shortly before its deadline. If this response window is shorter than the agent’s observation–reasoning–action latency, the agent cannot respond in time. Taken together, these findings show that turn-based agents can compensate for perception or action misalignment in isolation through observation histories or action sequences, but struggle with coupled real-time perception and action unless the dynamics are reproducible and retries allow experience transfer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
GUI Agents, Computer-Using Agents, Real-Time Interaction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_acUhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces RACE-Bench, which categorizes real-time GUI interactions into four task types according to perception and action requirements: continuous perception, anticipatory action, reproducible real-time interaction, and stochastic real-time interaction. By combining historical-frame access with within-turn action sequences, the paper analyzes the capability limitations of turn-based agents. The experiments primarily examine which tasks benefit from each of these two interface extensions and whether retries that retain prior experience help agents complete tasks that couple perception and action, with particular attention to difficulties commonly encountered by the evaluated systems on stochastic real-time tasks.

Critical strengths
  1. The research question has practical significance, and the task taxonomy helps clarify the capabilities required for real-time interaction. GUI agents need to acquire information and act promptly while their environments continue to change. The paper separately discusses continuous perception, action timing, and their coupling under reproducible and stochastic dynamics, providing a relatively clear framework for understanding the applicability of turn-based interaction. This taxonomy helps identify the capability requirements of different real-time tasks, extending the evaluation beyond overall task success rates.

  2. The interface ablation design is relatively clear, and the experiments provide valuable descriptive results. The paper introduces historical-frame access and within-turn action sequences separately, then combines them to compare the effects of these interface capabilities on different tasks. The results show that historical-frame access benefits perception tasks and action sequences benefit timed-action tasks, while these extensions remain limited on stochastic real-time tasks. These observations help clarify which interaction constraints turn-based agents can compensate for and under what conditions difficulties persist.

Critical weaknesses
  1. The analysis of response latency during actual tasks remains insufficient, limiting the interpretation of the core results. The paper uses response-latency constraints to explain low success rates on stochastic real-time tasks. Although it provides separate reaction-time measurements and a theoretical latency decomposition, it does not systematically report each model's end-to-end response latency during the actual evaluations or adequately distinguish the proportions of failures attributable to response timeouts and decision errors. Moreover, the environment continues running during historical-frame queries, while querying frames and executing actions require separate responses. The effects of additional tool interactions and model calls on evaluation results therefore remain unclear. Without these analyses, it is difficult to determine whether the observations arise from model decision-making ability, response speed, or their combined effects, and the evidence for explaining specific failure mechanisms from low success rates remains insufficient.

  2. The lack of independent repeated evaluations leaves the stability of the main results insufficiently verified. In the main model evaluation, only one run is retained for each model/configuration on each task. Up to three retries within a run, retaining experience, assess whether the agent can use prior experience to complete the task; they cannot substitute for independent repetitions under the same settings. Chance events and run-to-run variability may affect success rates, particularly on stochastic real-time tasks. The paper states that these results are descriptive statistics. Nevertheless, without independent repeated evaluations, it remains difficult to establish whether performance differences between configurations and models are stable and whether stochastic real-time tasks consistently yield low scores.

Questions for authors
  1. To what extent does actual response latency contribute to failures on stochastic real-time tasks? Could you report each model's end-to-end response latency during the actual evaluations and further distinguish failures caused by response timeouts from those caused by decision errors? Do the additional tool interactions and model calls required for historical-frame queries noticeably affect task performance? Could you compare models with similar latency or adjust task response windows to test whether the main results remain consistent and clarify the respective roles of decision-making ability and response speed?

  2. Do the main experimental conclusions hold consistently under independent repeated evaluations? Could you conduct independent repeated task evaluations under the same model and configuration settings, clearly distinguishing these repetitions from retries that retain experience within a run? In particular, for stochastic real-time tasks, could you verify whether performance differences between configurations and models, as well as consistently low overall success rates, can be reliably reproduced?

8 comments
AuthorsOpenReview ↗

Response to Reviewer acUh [Critical]

Dear Reviewer acUh,

Thank you for your careful review. We are glad that you find the research question practically significant and the task taxonomy a clear framework for when turn-based interaction applies (S1), and the interface ablation clear and its results valuable (S2).

So that we discuss the same argument, we first clarify how the paper argues for its capability boundary and how this argument attributes the failures, which our writing left unclear. We then show how much the main configuration varies under independent repetition, and close with your remaining points, which are marked [Optional] so that you can read them as time allows.

C1. [Critical] Unclear logic of the argument for the capability boundary (W1, Q1)

"The paper uses response-latency constraints to explain low success rates on stochastic real-time tasks."

Thank you for stating your reading so plainly, as it showed us exactly where the paper misleads. Because our abstract says that "D tasks reveal a timing constraint" and our conclusion that D "remains a turn-based boundary because execution cannot branch on new observations", the paper reads as if the timing constraint were an explanation fitted to the low success rates. What we wish to clarify is that the argument runs the other way: the boundary is derived first, from Eq. 1, and the tasks were designed to test it. Eq. 1 states that a response which depends on new information cannot arrive in time when the window from that information to the deadline is shorter than the agent's latency. The low success rates on D tasks are therefore a prediction that the experiments confirm, not the source of the boundary (G3 sets out the full argument).

The same derivation attributes each failure by where it occurs, without measuring failures one by one. In the table, A to D are the paper's four task types (continuous perception, anticipatory action, reproducible and stochastic real-time tasks), Video is the interface with historical-frame access, and Anticipate the one with within-turn action sequences; TRACE has both.

Where the failure occursWhat explains it
First attempts at C tasks (Double Jump aside, see E6), and D tasks with any of the paper's four interfacesResponse time: the information needed for the action appears only shortly before its deadline, and every model's end-to-end latency, which we measured for this rebuttal (E4), is far longer
A tasks with Video, B tasks with Anticipate, and C tasks within three attempts with TRACEDecisions and execution: the interface lets the agent take in the information long before the action is due (on C tasks, from an earlier attempt, as their dynamics repeat)

Latency is therefore sufficient to explain all observed failures on D tasks with the paper's four interfaces: the response window there is about as wide as the action-timing widths of the tasks, at most 1.5 s (Figure 2c), whereas during the evaluations no turn of any model took less than 2.2 s, and the median turn took 14.7 to 85.2 s, depending on the model (see the table in W1.3). The models can differ only in the second row, and they do, whatever their speed: GPT-6 Astra and MiniMax-M3 are among the three fastest models on both of our latency measures (W1.3), yet within three attempts they solve 82.4% and 0% of the C tasks, and the five models that are slower than both on both measures solve 52.9% to 76.5% (G7 compares the models further). G4 gives the derivation of this attribution, and we will reword the abstract and the conclusion accordingly in the camera-ready version.

C2. [Critical] Limited evidence for the soundness of the results (W2, Q2)

"The lack of independent repeated evaluations leaves the stability of the main results insufficiently verified."

Thank you for asking for this; the paper is stronger with it. We repeated the main configuration, DeepSeek with TRACE, three more times on all 69 tasks, each run independent of the others (E3). The runs use DeepSeek-V4.1-Flash-1017, which on 17 October replaced the version used in the paper (see Q2.2). On A, B and C, success within three attempts varied across the runs with a standard deviation of at most 8.3 points, whereas the gains on which the conclusions rest, from Video on A, Anticipate on B and TRACE on C, are 29 to 75 points, several times this variation. D stayed low in every run: within three attempts, it averaged 20.4% with a standard deviation of 16.0 points and never exceeded 38.9%, whereas seven external human participants averaged 86.5% with a standard deviation of 8.4 points (E5).

AuthorsOpenReview ↗

Response to Reviewer acUh [Optional] (1/4)

Although ICLR asks for brevity, we believe that every concern in your review deserves an answer, so the rest of this response goes through your review point by point, quoting each point before our answer. Please feel free to read only the points on which a concern remains, or to ask us directly instead. We thank you for the time and effort that your review took: each of your concerns is valuable to us, and we hope to resolve all of them.

W1. [Optional] Limited analysis of response latency

W1.1 Limited analysis of latency during the tasks

"The analysis of response latency during actual tasks remains insufficient, limiting the interpretation of the core results."

Every model is far slower than our tasks allow: during the tasks, a turn took 14.7 to 85.2 s at the median, depending on the model, and never less than 2.2 s, whereas the action-timing widths of our tasks are at most 1.5 s (W1.3 gives the measurements for each model). With a margin this wide, a finer analysis of latency would not change how the core results are interpreted.

W1.2 Unclear logic of the argument for the capability boundary

"The paper uses response-latency constraints to explain low success rates on stochastic real-time tasks."

See C1: the argument runs the other way, as the boundary is derived first and the tasks were designed to test it.

W1.3 Missing end-to-end latency of each model

"Although it provides separate reaction-time measurements and a theoretical latency decomposition, it does not systematically report each model's end-to-end response latency during the actual evaluations"

Thank you for this request. During the evaluations themselves, the trajectories of the main experiment give the time of every turn, from one screenshot to the next: the median turn took 14.7 to 85.2 s, depending on the model, and no turn took less than 2.2 s. As a second measure, we ran all eight models on the Human Benchmark reaction-time test (https://humanbenchmark.com/tests/reactiontime, three trials per model) in the same virtual machines and configurations as the main experiment; from the screen turning green to the click, they took 14 to 228 s on average. For each model with TRACE, in seconds (E4 gives the full tables):

ModelMedian turn during the evaluationsReaction test
GPT-6 Astra14.714.2
MiniMax-M326.217.1
Claude Fable 5.137.827.7
DeepSeek (turns: version of the paper; reaction test: version 1017)18.129.2
Kimi K367.449.2
GLM-5.3-Flash70.793.0
Qwen3.8-Max55.3112.6
Gemini 3.8 Flash85.2227.7

W1.4 Unclear proportions of timeouts and decision errors

"or adequately distinguish the proportions of failures attributable to response timeouts and decision errors."

See C1, whose table distinguishes the two by where the failure occurs.

W1.5 and W1.6 Unclear effect of extra tool calls for frame queries

"Moreover, the environment continues running during historical-frame queries, while querying frames and executing actions require separate responses."

"The effects of additional tool interactions and model calls on evaluation results therefore remain unclear."

Thank you for raising this. The extra responses cost the agent nothing it could otherwise use: without the tool, it would still need a full turn to see and act, which already takes far longer than our tasks allow, whereas with it, on the tasks where the tool applies, the agent no longer needs to observe in real time. The ablation in Table 1 shows this: Video adds only the frame tool to Vanilla, the plain turn-based interface, and with it DeepSeek solves 81.8% rather than 18.2% of the A tasks on the first attempt, while B, C and D change by at most one task. On a second model, GLM-5.3-Flash, adding the frame tool, to Vanilla or to Anticipate, lowers success on no task type, on the first attempt or within three attempts (E2, the interface ablation on a second model).

AuthorsOpenReview ↗

Response to Reviewer acUh [Optional] (2/4)

W1 (continued). [Optional] Limited analysis of response latency

W1.7 and W1.8 Unclear roles of decision-making and response speed

"Without these analyses, it is difficult to determine whether the observations arise from model decision-making ability, response speed, or their combined effects,"

"and the evidence for explaining specific failure mechanisms from low success rates remains insufficient."

Decision-making and response speed can be told apart, because the two never act together (see the table in C1). In its first row, the observations arise from response speed: the needed information appears only shortly before the deadline, so no model responds in time, whatever its decisions. In its second row, they arise from decision-making ability: the interface gives every model the information in advance, so speed no longer matters, and the models differ in what they decide. The failure mechanism is also not inferred from the low success rates. It follows from Eq. 1 and the task design.

W2. [Optional] Missing independent repeated evaluations

W2.1 Limited evidence for the soundness of the results

"The lack of independent repeated evaluations leaves the stability of the main results insufficiently verified."

See C2.

W2.2 Single retained run per configuration

"In the main model evaluation, only one run is retained for each model/configuration on each task."

This is common practice: most entries on the OSWorld-Verified leaderboard also report a single run (94 of the 110 with a score in November 2026; https://osworld-v1.xlang.ai/). To show how much a single run of the main configuration can vary, we added the three independent runs of C2.

W2.3 Retries in place of independent repetitions

"Up to three retries within a run, retaining experience, assess whether the agent can use prior experience to complete the task; they cannot substitute for independent repetitions under the same settings."

We agree, and the three new runs share no history. Their first attempts therefore form three tries without memory, which separates what memory adds from simply trying more often: within three attempts, success on C is 5.9% without memory and 47.1% with it on average, and each run with memory solves significantly more C tasks than this control, whereas memory does not solve significantly more D tasks (E3, the three new runs with this memory-cleared control).

W2.4 Unknown effect of chance and run-to-run variability

"Chance events and run-to-run variability may affect success rates, particularly on stochastic real-time tasks."

They do, and C2 measures by how much for the main configuration.

W2.5 Results limited to descriptive statistics

"The paper states that these results are descriptive statistics."

We agree that the sentence in Appendix D.1 calling our results "descriptive task averages" sets no scope. In the camera-ready version, we will rewrite it to state that each average comes from one run, and add an exact test for the comparison behind each of the paper's four conclusions, one per task type:

  • Video raises success on A tasks: with DeepSeek, from 8 to 20 of the 22 A tasks within three attempts (36.4% to 90.9%; exact McNemar test over the paired tasks, p = 0.002).
  • Anticipate raises success on B tasks: from 1 to 10 of the 12 B tasks within three attempts (8.3% to 83.3%; same test, p = 0.004).
  • TRACE solves C tasks through retained experience rather than extra attempts: the gain from memory in W2.3 is significant in each of the three new runs (same test, p = 0.031, 0.021 and 0.016). Across the eight models of the paper, C reaches 58.8% within three attempts against 8.6% expected from three independent first attempts (1-(1-r)^3 with their pooled first-attempt rate r = 4/136, see E3).
  • None of the four interfaces reliably lifts D tasks above the random reference: in the run of the paper, neither DeepSeek with any of the four interfaces nor any of the eight models with TRACE solves significantly more D tasks within three attempts than the random policy (with DeepSeek, 3, 4, 4 and 3 of the 18 tasks, against 2.7 expected; one-sided exact test of the number of tasks solved against the per-task random references of Table 10, p > 0.25); W2.7 applies the same test to the repetitions.

Beyond these tests, the gains on A, B and C are far larger than the variation between the three runs of DeepSeek with TRACE, and they reappear on a second model (see W2.6).

Reviewer_Pp6dOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the limitations of turn-based GUI agents in real-time environments, motivated by the contrast between overlapping human perception and action and the sequential observe–reason–act loop commonly used by agents. The authors introduce RACE-Bench, which organizes atomic interactions into continuous perception, anticipatory action, reproducible real-time coupling, and stochastic real-time coupling tasks. Within a shared environment, they compare a screenshot-based agent, an agent with historical video access, an agent supporting timed action sequences within a turn, and TRACE, which combines both extensions. An experience-retaining retry protocol examines how these mechanisms compensate for interaction constraints. The paper further characterizes a conditional timing limit through the relationship between the window from information availability to the action deadline and the necessary response latency. Cross-model evaluations, human references, trajectory analyses, and deployment on a physical computer examine which bottlenecks can be alleviated by observation histories, precommitted actions, and experience reuse, and where these mechanisms remain insufficient for stochastic real-time interaction.

Critical strengths
  1. The paper provides a clear and practically relevant diagnostic framework by separating missed perceptual information, action-timing constraints, and their coupling under reproducible or stochastic dynamics. Atomic tasks reduce confounding from long-horizon planning and complex rule learning, allowing the analysis to focus on when information becomes available and when an action must occur. This offers a useful perspective beyond aggregate task success.

  2. The four configurations compare historical video access and within-turn action sequences while holding the underlying model fixed. This helps reveal the constraints addressed by each extension and the strategies their combination enables. Distinguishing first-attempt success from success after experience-retaining retries, together with concrete probe-then-execute trajectories, also provides interpretable evidence about how compensation can occur.

  3. The evaluation covers multiple models and includes physical-computer deployment and human references, demonstrating the phenomenon across different system configurations. External participants are reported separately from participants familiar with benchmark development, and the paper carefully distinguishes action-timing tolerances, information-exposure durations, and reaction latency. The appendices provide extensive task inventories, prompts, tool restrictions, random references, and per-task outcomes, making the evaluation protocol and its scope easier to inspect.

Critical weaknesses
  1. The central latency boundary is not sufficiently connected to the observed failures through quantitative evidence. Section 3.3 and Appendix G correctly state that an information-dependent response cannot arrive in time when the response window is shorter than the necessary observation–decision–execution latency. However, Figure 2 primarily reports action-timing tolerances, which the paper explicitly distinguishes from information-to-deadline windows. The physical reaction test provides only a few end-to-end observations for one configuration and does not establish the necessary latency bounds for the evaluated models. The study lacks per-task measurements linking critical information availability, usable observations, decisions, and action effects, as well as interventions that vary response windows or system latency while holding the task otherwise fixed. Consequently, low D-task success does not adequately distinguish latency limitations from incorrect decisions or execution errors. The paper's central interpretation in terms of a capability boundary therefore remains insufficiently validated.

  2. The comparisons mainly characterize the limitations of a fixed-sequence execution interface, without sufficiently identifying the minimal mechanism needed to cross that boundary. TRACE cannot branch on new observations during a submitted sequence, so a response to an unexpected event must await another model decision. Prior work such as AAPT instead prepares conditional policies in advance and uses a lightweight observer to route live observations to a branch. The authors cite these approaches and appropriately place them outside the evaluated interface, but the absence of comparisons with precompiled branching, short-chunk replanning, or faster decision paths limits the mechanistic conclusion. The experiments do not isolate the respective contributions of full-model inference latency and the executor's lack of feedback. Comparisons that preserve the task, available visual information, and underlying model while changing the event-time decision path would provide a more informative characterization of the boundary.

  3. Improvements across attempts do not sufficiently isolate the contribution of experience transfer. The gap between first-attempt and multiple-attempt C-task success is consistent with learning timing from an earlier trajectory. However, comparison with a specified random policy cannot replace a matched evaluation of the same agent with its history cleared. Appendix C.2 explicitly acknowledges that the comparison does not separate retained observations, strategy revision, and other changes across attempts. The results therefore do not quantify how much of the retry benefit specifically depends on reusable experience under reproducible dynamics. Moreover, only one retained run is available per task and configuration, and seeds were not recorded, limiting conclusions about stochastic-task success probabilities and the stability of configuration differences. Matched memory-retention controls and controlled repetitions are needed to substantiate the proposed connection between reproducibility and retry-based compensation.

Questions for authors
  1. Historical video access requires the model to select timestamps and retrieve a limited number of frames per query. How does the agent locate brief events, and how often does its retrieval strategy miss relevant information that is present in the recording? Could the authors compare adaptive retrieval with uniformly sampled frames under matched frame counts or visual-input budgets, to clarify the roles of information retention, temporal localization, and additional visual input in the A-task gains?

  2. The paper reports that MiniMax often describes a multi-step plan but submits only one action, and other trajectories contain mistakes in keypress semantics. Could the authors report valid tool-call rates, actual submission rates for action sequences, and execution success when a verified sequence is supplied? These diagnostics would help distinguish protocol adherence and understanding of control primitives from the real-time perception–action capabilities targeted by the benchmark.

  3. Section 5.4 and Appendix E.4 describe a D3 mouse-sweep success as a chance hit, although the reported motion could not reach the target under the game mechanics. Which environment event or collision condition triggered that success, and can it be verified through state logs or frame-by-frame replay? How do the authors establish that this was a legitimate success without an informed response, rather than a discrepancy between the success check and the visible interaction?

9 comments
AuthorsOpenReview ↗

Response to Reviewer Pp6d [Critical] (1/2)

Dear Reviewer Pp6d,

Thank you for your careful and constructive review. We are glad that you find the diagnostic framework clear and practically relevant (S1), that the comparison of the four configurations with the model held fixed helps reveal what each extension addresses (S2), and that the evaluation is broad, carefully distinguished and easy to inspect (S3).

We first show how the boundary connects to the observed failures, a connection that our writing left unclear. We then report the two experiments you asked for, an interface that branches on new observations and a control with memory cleared, and close with your remaining points, which are marked [Optional] so that you can read them as time allows.

C1. [Critical] Limited quantitative connection between the boundary and the failures (W1)

"The central latency boundary is not sufficiently connected to the observed failures through quantitative evidence."

"Consequently, low D-task success does not adequately distinguish latency limitations from incorrect decisions or execution errors."

Thank you for this point, and for confirming that Sec. 3.3 and Appendix G "correctly state" the boundary. On D tasks with the paper's four interfaces, the information-to-deadline window is at most about 1.5 s, whereas no turn of any evaluated model took less than 2.2 s, so latency is sufficient to explain every observed failure there, however good the decisions. The latency is measured in the main experiment itself, over 6,608 turns; with TRACE, the median turn takes 14.7 to 85.2 s depending on the model (E4, our latency measurements). The window is about as wide as the action-timing tolerance of Figure 2 on these tasks (W1.3 explains why).

Decisions and execution are what the scores measure once an interface lifts this timing condition, and there the same models succeed far more often: with TRACE, the eight models solve on average 58.8% of the C tasks within three attempts but 16.0% of the D tasks (82.4% and 11.1% for GPT-6 Astra). Conversely, we reviewed by hand the 40 successes that could contradict the boundary, namely every first-attempt success on C and every success on D of the eight models with TRACE in Table 1 and of the three runs of E3, and none was a timely response to its event (see E6). We will state this attribution in Sec. 5 of the camera-ready version.

AuthorsOpenReview ↗

Response to Reviewer Pp6d [Critical] (2/2)

C2. [Critical] Missing comparison with interfaces that branch on new observations (W2)

"The authors cite these approaches and appropriately place them outside the evaluated interface, but the absence of comparisons with precompiled branching, short-chunk replanning, or faster decision paths limits the mechanistic conclusion."

Thank you for this suggestion, which led to the most informative new result of this rebuttal: testing such an interface tightens the boundary.

We turned the idea of AAPT [1] into a lightweight observer that the agent can start and stop (G1 calls it a subagent). The observer runs asynchronously: it keeps taking screenshots and acting on what it sees in its own loop while the agent goes on thinking. When starting it, the agent specifies what to watch for and what to do, either as conditions in plain language that a fast vision model (Jev 1.14) matches against each screenshot, or as a Python script that reads the pixels itself (G1 describes the two tools). This one interface covers the three paths that you list: the conditions or the script are precompiled branches; each observer runs one short chunk, and the agent can replan by stopping it and starting a new one; and the event-time decision is made by the fast vision model, in 0.72 s on average, or by the script, instead of by the agent. We tested both kinds once each on all 18 D tasks with DeepSeek, keeping the tasks, the visual information and the model fixed and changing only the event-time decision path, as you propose.

An observer enables the agent to answer an event in time once it has seen it, but not on a first encounter. DeepSeek solved 12 of the 18 D tasks within three attempts with a script-driven observer and 3 with a Jev-driven one, but only 1 on the first attempt with each (E1 lists the tasks). Eq. 1 explains why. The agent must write the observer before starting it, and it can do so only after seeing the event: before that, it knows neither what the event looks like on screen nor how to answer it. Unlike a fixed action sequence, an observer waits for the event and reacts to it whenever it comes, so an observer written after one attempt still works in the next, even though the event does not repeat exactly. Once the agent has seen the event, it has until the event comes again to start the observer, so the information-to-deadline window spans attempts instead of a fraction of a second. On a first attempt, the window is still the short time between the event and its deadline. The two first-attempt successes agree with this: neither answered in time an event that the agent had not seen before (see E1). Fewer tasks therefore lie beyond the boundary than the paper states, while Eq. 1 is unchanged: a task lies beyond it only when its event is met for the first time and must be answered at once. We will revise the abstract and the conclusion accordingly in the camera-ready version.

[1] Dong et al. Why are GUI agents correct but late? Decode on the decision-time critical path, tested with pre-compiled policy trees. arXiv:2607.28399, 2026.

C3. [Critical] Limited isolation of the contribution of experience transfer (W3)

"Matched memory-retention controls and controlled repetitions are needed to substantiate the proposed connection between reproducibility and retry-based compensation."

Thank you for asking for both; the paper is stronger with them. Below are the results.

  • Measured control: we ran the main configuration, DeepSeek with TRACE, three more times on all 69 tasks, with DeepSeek-V4.1-Flash-1017, which on 17 October replaced the version used in the paper. Because the runs share no history, their first attempts give each task three attempts without any memory, the same budget as one run that keeps its history. On C, success within three attempts is 5.9% without memory and 47.1% with it, and in every run memory solved significantly more tasks than the control (exact McNemar test over the paired tasks, p = 0.031, 0.021 and 0.016). On D, it is 11.1% without memory and 20.4% with it, a difference that is not significant in any run (E3 gives each run).
  • Estimated control for the eight models of the paper, which have one run each: with a first-attempt rate r, pooled over the models, three attempts without memory succeed with probability 1-(1-r)^3. On C, this gives 8.6%, against the 58.8% that the models reach with retained history; on D, 17.6%, against 16.0% (E3).

The controls thus support the connection that the paper proposes: retries help through reusable experience where the dynamics repeat, and not where they change.

AuthorsOpenReview ↗

Response to Reviewer Pp6d [Optional] (1/6)

Although ICLR asks for brevity, we believe that every concern in your review deserves an answer, so the rest of this response goes through your review point by point, quoting each point before our answer. Please feel free to read only the points on which a concern remains, or to ask us directly instead. We thank you for the time and effort that your review took: each of your concerns is valuable to us, and we hope to resolve all of them.

W1. [Optional] Limited quantitative connection between the boundary and the failures

W1.1 Unclear logic of the argument for the capability boundary

"The central latency boundary is not sufficiently connected to the observed failures through quantitative evidence."

See C1.

W1.2 Correct statement of the boundary

"Section 3.3 and Appendix G correctly state that an information-dependent response cannot arrive in time when the response window is shorter than the necessary observation-decision-execution latency."

Thank you for confirming this. It is the first step of our argument (G3 sets out all three): the boundary rests on this derivation and not on the scores.

W1.3 Unclear relation between action-timing tolerances and information-to-deadline windows

"However, Figure 2 primarily reports action-timing tolerances, which the paper explicitly distinguishes from information-to-deadline windows."

You are right that the two differ, and thank you for showing us that the paper does not say when they coincide. The information-to-deadline window W depends on when the needed information becomes available, which changes with the interface and with earlier attempts, whereas the action-timing tolerances belong to the task. In B tasks, the tolerances are short, but all information is given before the agent presses Start, so W is wide. On first attempts at C and D tasks, with the exception of the C task Double Jump discussed in E6, and on every attempt at D tasks without a lightweight observer, the information is complete only when the action window opens, so W is about as wide as the action-timing tolerances. In Random Parry (D28; https://anonymous.4open.science/w/demo-games-7F3AC9/d/D28_Random_Parry/index.html), for example, the blade's appearance does not yet tell the agent when to parry. A reply such as "wait 1.3 s, then parry" would start its wait only once the reply has been written, and the agent cannot know in advance how long writing it will take. Only the cue that lights up as the blade comes within reach completes the information, and an immediate parry is then the only option. On C tasks within three attempts, the dynamics repeat, so the information can come from an earlier attempt, and W is far wider than the action-timing tolerances. Figure 2 therefore reports the action-timing tolerances, which are fixed for each task; on first attempts at C and D tasks (Double Jump aside) and on D tasks without an observer, they stand in for W. We will add this explanation to Sec. 3.3 in the camera-ready version.

W1.4 Missing end-to-end latency of each model

"The physical reaction test provides only a few end-to-end observations for one configuration and does not establish the necessary latency bounds for the evaluated models."

Thank you for this request. We now report the latency of all eight models in two ways (E4 lists both for each model). C1 gives the first: the turns of the main experiment itself, from one screenshot to the next. The second is the Human Benchmark reaction-time test that the paper uses for people, run in the same virtual machines, on which the models took 14 to 228 s on average. A measurement cannot prove a lower bound, but the margin leaves little room: the fastest turn of any model is still slower than the widest action-timing tolerance of 1.5 s, and the median turn of every model is about ten times slower or more.

Reviewer_vgZzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds RACE-Bench, 69 real-time mini-games added to OSWorld and split into four groups: continuous perception (A), anticipatory action (B), perception and action coupled under repeatable dynamics (C), and perception and action coupled under stochastic dynamics (D). It compares four interfaces. Vanilla sees one screenshot and sends one action per turn, Video can retrieve past frames, Anticipate can send an ordered action sequence with waits in one response, and TRACE has both extensions. The full ablation uses DeepSeek v4.1 Flash. Seven more models, including GPT-6 Astra, Claude Fable 5.1 and Gemini 3.8 Flash, are run with TRACE. A physical-computer deployment and human participants serve as references. The authors conclude that past frames and action sequences each address one kind of limitation, that retries help when the dynamics repeat, and that stochastic coupled tasks stay hard for turn-based agents.

Critical strengths
  1. The implementation is substantial and can be partly checked. The anonymous OSWorld repository has all 69 task pages, 69 task configurations, 32 model and interface configurations, and the Video and Anticipate interfaces. I re-aggregated the 69 per-task records from the physical deployment and got 27/69 first-attempt and 44/69 three-attempt successes (A 21/22, B 11/12, C 10/17, D 2/18), the same as the paper. The authors' note about a resumed 61-task batch plus eight earlier records also matches the release.

  2. The four-way split is easy to read as a diagnostic, and the study covers more than one model family. On the main model the interface effects are large: A-category success within three attempts goes from 36.4% to 90.9% with Video, and B-category success goes from 8.3% to 83.3% with Anticipate. Proprietary models are included (TRACE overall success within three attempts is 72.5% for GPT-6 Astra, 71.0% for Claude Fable 5.1 and 68.1% for Gemini 3.8 Flash), and first-attempt and three-attempt results are reported separately.

Critical weaknesses
  1. The way tasks were selected, together with thin statistics, limits what the results say about a general capability boundary. Tasks were admitted with a compensability criterion: an A, B or C task had to be solvable by the matching interface extension, and D tasks kept response-relevant randomness. Acceptance testing checked this before any model was evaluated. So the A, B and C results largely confirm the admission rule, which weakens them as independent evidence and narrows how far they generalize beyond these interfaces. For D, the useful question is how many of the observed failures actually meet the timing condition (the response window is shorter than the agent's minimum latency) and how many come from ordinary perception or control errors. The paper does not show this. The statistics are also weak. Each task and configuration has one retained run, seeds were not recorded, and the stochastic tasks I inspected draw their events from Math.random with no seeded replay, so matched conditions and run-to-run variation cannot be checked. With 18 D tasks, one task is worth 5.6 points, and the D differences between configurations amount to one to four tasks. Several results are at or below the paper's own random reference: Vanilla reaches 18.8% overall within three attempts against 30.8%, and DeepSeek TRACE reaches 16.7% on D against 14.8%. The C gain from 0% to 58.8% mixes retained observations, strategy changes and extra attempts, and the authors say a memory-clearing control is missing. The interface ablation covers one model, and the main human reference is two external participants (two participants familiar with development are reported separately).

  2. The released evaluation code has concrete reproducibility and record-keeping problems. The runner imports mm_agents.os_symphony at module level, and that module is missing from the release. Even run_multienv.py --help fails, and 4 of 164 offline tests I selected fail for this reason (the other 160 pass). The resume logic is a bigger problem. If the trajectory audit fails after a game has finished, no result.txt is written, the resume routine treats the task as unfinished, deletes its files (including result.json and trajectory.jsonl) and plays the game again. I reproduced this in isolation for both a finished success and a finished failure. This can replace a stochastic outcome and erase the local record of the original one. It does not show that any reported score went up, but the repositories I inspected do not include archived per-task outcomes or full trajectories for the main OSWorld experiments, so I could not check whether this path affected them.

Questions for authors
  1. For representative D tasks, can the existing traces line up when the needed information first appears, the deadline, and when the action took effect? How many D failures meet the timing condition, and how many come from other perception or control errors? Results for an interface that can react to new observations inside a committed sequence would also help separate the effect of latency from the effect of the no-branching rule.

  2. Do the existing records allow comparisons under matched random conditions, or a retain-versus-clear-history comparison with the same three-attempt budget? If not, please narrow the boundary and memory-attribution claims, and say which category-level differences exceed the random reference by more than a few tasks.

  3. Can you provide a self-contained release, or pinned upstream assembly steps, that fixes the missing mm_agents.os_symphony import? Please identify the code snapshot behind the reported OSWorld results and share the matching per-task outcomes and sanitized trajectories. How many finished games were discarded and replayed because the audit failed, and were the original records kept anywhere? Keeping the finished game and retrying only the audit would fix the problem.

  4. The released audit note for C1_Double_Jump says an Alt+Tab exposed an operator window and the agent later mentioned the operator's chat text. What was visible, and could it have helped with the task? Please share a sanitized record of this episode and explain how task-relevant observations were isolated in the physical deployment. The automatic NOT_CHEAT label does not settle this, and the note alone does not show cheating either.

  5. Real-time GUI performance depends heavily on inference latency, which can vary across GPU hardware, serving configurations (e.g., single-GPU, multi-GPU, or multi-node deployment), and API conditions. Although Appendix H acknowledges this dependency, the paper does not report controlled deployment or latency sensitivity experiments, nor per-model end-to-end latency distributions. To what extent could differences in deployment infrastructure affect the reported cross-model comparisons and capability-boundary conclusions? Could the authors report end-to-end response latency and evaluate at least one model under different deployment or controlled-latency settings to quantify this sensitivity?

9 comments
AuthorsOpenReview ↗

Response to Reviewer vgZz [Critical] (1/2)

Dear Reviewer vgZz,

Thank you for your thorough review, and for checking our release yourself: you re-aggregated the per-task records of the physical deployment, ran the tests and reproduced the resume path. We are glad that you find the implementation substantial (S1), the four-way split easy to read as a diagnostic, and the interface effects on the main model large (S2).

So that we discuss the same argument, we first clarify how the paper argues for its capability boundary and how the observed failures are attributed, which our writing left unclear. We then answer your concerns about the statistics and the release, and close with your remaining points, which are marked [Optional] so that you can read them as time allows; among them, Q1.3 reports the interface you asked for, which reacts to new observations and narrows the boundary.

C1. [Critical] Unclear logic of the argument for the capability boundary (W1, Q1)

"So the A, B and C results largely confirm the admission rule, which weakens them as independent evidence and narrows how far they generalize beyond these interfaces."

"For D, the useful question is how many of the observed failures actually meet the timing condition (the response window is shorter than the agent's minimum latency) and how many come from ordinary perception or control errors."

Thank you for raising these two points, which showed us that the paper does not say what role its experiments play. Because the paper presents the boundary (Sec. 3.3), the task design (Sec. 4) and the results (Sec. 5) separately, the boundary can appear to be inferred from the results, and results on tasks built from it would then only confirm the construction. What we wish to clarify is that the argument runs the other way: the boundary is derived first, and the experiments test its predictions.

The boundary follows from the turn-based loop itself (Sec. 3.3 and Appendix G), so it holds independently of the experiments. The tasks were then designed from the definitions of the four task types and checked by people against these definitions, without running any agent, and no task was added, removed or moved to another type because of a model result. That a correct strategy exists under the matching interface is thus a property of the design, which ensures that a failure cannot be blamed on an unsolvable task. What the design cannot ensure, and the experiments test, is whether agents behave as the boundary predicts: that they fail without the matching extension, succeed with it, solve C tasks mainly after retries, and stay low on D tasks. Each of these predictions could have failed, and the results bear each of them out (Table 1, and our new experiments E2 to E5). The results on A, B and C are therefore not offered as independent discoveries from which the boundary is inferred, but as tests that it passed. We also reviewed by hand 40 unexpected successes, namely every first-attempt success on C and every success on D of the eight models with TRACE in Table 1 and of the three runs of E3, which are the kind of result that could contradict the boundary, and none was a timely response to its event (E6).

The same derivation answers your question on D. Under the paper's four interfaces, every attempt at a D task meets the timing condition, so timing is sufficient to explain every observed failure, and we do not split the failures by cause. On these tasks, the information needed for the action is complete only when the action window opens (in Random Parry, the cue lights up 0.5 s before the blade strikes; see Q1.1), so the response window is about as wide as the action-timing widths of Figure 2c, at most 1.5 s. In the main experiment, no turn of any model took less than 2.2 s, and the median turn of each model took 14.7 to 85.2 s (E4, our latency measurements). An information-dependent response therefore could not arrive in time in any attempt, however good the perception and the control. Where an interface lifts the timing condition (A tasks with Video, B tasks with Anticipate, C tasks within three attempts with TRACE), the remaining failures are decision and execution errors, including the perception and control errors that you mention. There, most of the same models succeed: GPT-6 Astra, for example, solves 82.4% of the C tasks within three attempts but 11.1% of the D tasks, with the same interface.

G3 sets out the full argument and G4 the attribution. Q1.3 reports how far both generalize beyond the four interfaces: with an interface that reacts to new observations, DeepSeek solves 12 of the 18 D tasks within three attempts but only 1 on the first, so the boundary narrows to events that are met for the first time and must be answered at once. In the camera-ready version, we will state this argument in Sec. 3.3 and the attribution in Sec. 5, and reword the description of task construction (Sec. 1 and Appendix A.1), which reads as a selection.

AuthorsOpenReview ↗

Response to Reviewer vgZz [Critical] (2/2)

C2. [Critical] Limited evidence for the soundness of the results (W1, W2, Q2, Q3)

"The statistics are also weak."

"The released evaluation code has concrete reproducibility and record-keeping problems."

Thank you for the care with which you examined our evidence. Each gap that you named led to a new result, and these results confirm the conclusions that the paper draws for its four interfaces. We repeated the main configuration three more times, added the memory-cleared control, a second model and five more participants, and fixed and completed the release:

  • Repetitions: across three more independent runs of DeepSeek with TRACE, success within three attempts has a standard deviation of at most 8.3 points on A, B and C, whereas the gains on which the conclusions rest are 54.5 points on A with Video, 75.0 on B with Anticipate, and 29.4 to 52.9 on C with TRACE over the other three interfaces; on D, it is 16.0 points, more than any difference among the four interfaces in Table 1 (E3). These runs use DeepSeek-V4.1-Flash-1017, which on 17 October replaced the version used in the paper.
  • Memory-cleared control: taking the first attempt of each of the three independent runs gives every task three tries without memory. With this same budget, success on C is 5.9% without memory and 47.1% with it (the mean of the three runs), and each run with memory solves significantly more C tasks than this control, whereas memory does not solve significantly more D tasks (E3).
  • Second model: we ran all four interfaces anew with GLM-5.3-Flash, and each gain on A, B and C reappears (E2).
  • Human reference: with five more external participants, seven in total, every participant solves at least 72.2% of the D tasks within three attempts, whereas no agent run with the paper's four interfaces exceeds 38.9% (E5).
  • Code and records: the release now runs, as we checked by setting it up from scratch on a fresh server; the resume path keeps the finished game and retries only the audit, as you suggest; and the per-task outcomes and text trajectories of all 759 task runs of the main experiment are released, as are the per-task results of every new experiment, so every result can be checked (W2 and Q3 give the details, including the 32 games that the old resume path replayed).
AuthorsOpenReview ↗

Response to Reviewer vgZz [Optional] (1/6)

Although ICLR asks for brevity, we believe that every concern in your review deserves an answer, so the rest of this response goes through your review point by point, quoting each point before our answer. Please feel free to read only the points on which a concern remains, or to ask us directly instead. We thank you for the time and effort that your review took: each of your concerns is valuable to us, and we hope to resolve all of them.

W1. [Optional] Limited generality of the results for a capability boundary

W1.1 Unclear generality of the capability boundary

"The way tasks were selected, together with thin statistics, limits what the results say about a general capability boundary."

See C1 for the tasks, which were designed from the definitions of the four task types and not selected by results, and C2 for the statistics. As for generality, the boundary is derived from the turn-based loop, so it does not depend on the sample of tasks.

W1.2 and W1.3 Unclear role of the admission criterion

"Tasks were admitted with a compensability criterion: an A, B or C task had to be solvable by the matching interface extension, and D tasks kept response-relevant randomness."

"Acceptance testing checked this before any model was evaluated."

This describes the paper correctly, and we apologize that our wording suggests a filter. The criterion is the definition of the four task types, written as a requirement on the design: a task belongs to a type by the way it couples perception and action. An A task, for example, requires ongoing perception but no time-critical response, so whether its information can be recovered from the recording is a property of the task itself, which a person can judge by playing it. People then played each task and checked it against these definitions, without running any agent or interface (see C1). We will reword Sec. 1 and Appendix A.1 accordingly in the camera-ready version.

W1.4 Unclear logic of the argument for the capability boundary

"So the A, B and C results largely confirm the admission rule, which weakens them as independent evidence and narrows how far they generalize beyond these interfaces."

See C1, which answers both halves: the results are tests of predictions that could have failed, and an interface beyond the four of the paper narrows the boundary without contradicting it (Q1.3).

W1.5 and W1.6 Unclear attribution of failures to response time or to decisions

"For D, the useful question is how many of the observed failures actually meet the timing condition (the response window is shorter than the agent's minimum latency) and how many come from ordinary perception or control errors."

"The paper does not show this."

See C1: under the paper's four interfaces, every attempt at a D task meets the timing condition. You are right that the paper does not show this, and the camera-ready version will state it in Sec. 5.

W1.7 and W1.8 Single retained run and unrecorded seeds

"The statistics are also weak."

"Each task and configuration has one retained run, seeds were not recorded, and the stochastic tasks I inspected draw their events from Math.random with no seeded replay, so matched conditions and run-to-run variation cannot be checked."

The run-to-run variation can now be checked: within three attempts, three more independent runs of DeepSeek with TRACE give a standard deviation of at most 8.3 points on A, B and C and of 16.0 points on D (E3). The paper reported one run per configuration, as most entries on the OSWorld-Verified leaderboard do (94 of the 110 with a score in November 2026; https://osworld-v1.xlang.ai/). The events are indeed drawn from the browser's Math.random, which takes no seed, so there is none to record or replay, and we keep the games unseeded on purpose. With a fixed seed, the random events would repeat from one evaluation to the next, so a model trained on the benchmark could memorize them, and the D tasks would no longer test responses to events that cannot be known in advance. Matched random conditions are therefore not available, but the comparisons remain fair (see Q2.1), and the independent runs measure the effect of the randomness (G9 is our answer on seeds to all reviewers).

W1.9 Small differences between configurations on D

"With 18 D tasks, one task is worth 5.6 points, and the D differences between configurations amount to one to four tasks."

We agree, and the paper draws no conclusion from these differences. Its claim on D is the opposite: none of the four interfaces helps, so success "remains close to the random reference on D" (Sec. 5.1). The repetitions agree: across the three runs of DeepSeek, the standard deviation on D is 16.0 points, about as much as the largest difference in Table 1, where the configurations solve one to four D tasks: three tasks (16.7 points) between models and one task (5.6 points) between the four interfaces.

Comments on the paper

12 comments
AuthorsOpenReview ↗

General Response to All Reviewers [Critical] - part 1/4

Dear Reviewers and Area Chair,

We sincerely thank all of you for your time and constructive feedback. We are encouraged that the reviewers commonly recognized the strengths of our work, including our clear and practically relevant diagnostic framework (Pp6d Strength 1; acUh Strength 1; vgZz Strength 2), our clearly designed interface ablation with valuable results (Pp6d Strength 2; vgZz Strength 2; acUh Strength 2), our broad multi-model evaluation with a physical deployment and human references (Pp6d Strength 3; vgZz Strength 2), and our substantial and transparent implementation (vgZz Strength 1; Pp6d Strength 3). We are equally grateful for the weaknesses raised, which have made the paper better.

Since ICLR asks for brevity, [Optional] marks what can be skipped. And since the tasks are easier to understand by playing them than by reading about them, we have put them online for you to try: https://anonymous.4open.science/w/demo-games-7F3AC9/index.html.

In brief, our answers to the shared concerns are:

  • G1 Interfaces that branch on new observations: we gave the model access to a reactive subagent, in the spirit of AAPT [1], that watches the screen and acts on its own. With it, the model solves up to two-thirds of the D tasks once it has seen their event, but almost none on the first attempt. This narrows the boundary without changing Eq. 1 of the paper (i.e., a response cannot arrive in time when the window from information to deadline is shorter than the agent's latency).
  • G2 Soundness: a memory-cleared control, a second model, five more participants and the released records all support the results, and across three more independent runs of DeepSeek, the variation is far smaller than the gains on which the conclusions rest.
  • G3 Logic of the argument: the boundary is derived and holds independently of the experiments, the experiments test its predictions, and none of the 40 first-attempt successes on C and successes on D that we reviewed by hand was a timely response to its event.
  • G4 Attribution of failures: where the response window is shorter than the latency, timing is sufficient to explain the failures; where an interface separates information acquisition from time-critical execution, the remaining failures come from decisions and execution.
  • G5 [Optional] Latency: on a reaction-time test, all eight models in virtual machines take 14 to 228 s on average to respond, and during the tasks their median turn takes 14.7 to 85.2 s, far beyond the 1.5 s that our tasks allow at most.
  • G6 [Optional] Varying the windows or the latency: we keep the windows at human speed, because our question is which tasks people can do and turn-based agents cannot.
  • G7 [Optional] Latency and model differences: the differences come from decisions, not latency, as the task windows require human-level response speeds that none of the eight models comes close to (G5).
  • G8 [Optional] Per-task timelines: one timeline per task is not well defined, because when the needed information appears depends on the interface and on earlier attempts.
  • G9 [Optional] Random seeds: the random dynamics stay unseeded so that they cannot be memorized, and three independent runs of DeepSeek measure their effect instead.

[1] Dong et al. Why are GUI agents correct but late? Decode on the decision-time critical path, tested with pre-compiled policy trees. arXiv:2607.28399, 2026.

AuthorsOpenReview ↗

General Response to All Reviewers [Critical] - part 2/4

G1. [Critical] Missing comparison with interfaces that branch on new observations (Reviewer Pp6d W2; Reviewer vgZz W1, Q1)

We thank Reviewers Pp6d and vgZz for this suggestion. When writing the paper, we did not test such an interface, because we believed it would only move the real-time requirement from one agent to another. Your reviews made us realize that testing it tightens the boundary, so we built it and ran it, and the result makes our conclusion more rigorous.

We turned the idea of AAPT into a reactive subagent that the agent can start and stop. AAPT (Dong et al., 2026) [1] prepares conditional policies in advance and "uses a lightweight observer to route live observations to a branch" (Reviewer Pp6d). Since every action in our framework is a tool call, we gave the agent two more tools in TRACE (the released code calls the subagent an observer). create_observer starts an asynchronous subagent and returns at once, so the call does not block the agent: the agent goes on thinking and acting, while the subagent keeps taking screenshots and acting on what it sees in its own loop, much faster than the agent's turns. stop_observer stops it. When starting the subagent, the agent specifies (1) what to watch for, as branching conditions, and (2) what to do in each branch. It does so in one of two ways: a Jev-driven subagent receives the conditions in plain language and asks a fast vision model (Jev 1.14) which one each new screenshot matches, and a script-driven subagent receives a Python script that reads the pixels itself. We tested both on all 18 D tasks with DeepSeek-V4.1-Flash-1017 (E1).

A subagent enables the agent to answer an event in time once it has seen it, but not on a first encounter. In Eq. 1, the response window runs from when the information needed for a response appears to when the response must take effect. Starting the subagent is itself an action of the agent and only has to happen before the event comes again, so the window runs from when the agent first saw the event, usually in an earlier attempt, until the event comes again, far longer than one of its turns. On a first attempt, the window is still the short time between the event and its deadline. Accordingly, the agent solved 12 of the 18 D tasks within three attempts with a script-driven subagent and 3 of 18 with a Jev-driven one, but only 1 of 18 on the first attempt with each (E1). These results narrow the boundary without changing Eq. 1: a task now lies beyond it only when its event is met for the first time and must be answered at once. The original conclusion still holds for the four interfaces we evaluated, and we will revise the abstract and conclusion accordingly in the camera-ready version.

[1] Dong et al. Why are GUI agents correct but late? Decode on the decision-time critical path, tested with pre-compiled policy trees. arXiv:2607.28399, 2026.

AuthorsOpenReview ↗

General Response to All Reviewers [Critical] - part 3/4

G2. [Critical] Limited evidence for the soundness of the results (Reviewer vgZz W1, W2, Q2, Q3; Reviewer Pp6d W3; Reviewer acUh W2, Q2)

We thank the reviewers for their careful attention to the soundness of our results, which pushed us to make the work stronger. We took the following steps, each a standard way to test whether results are sound, and we found that all of the new results support the conclusions of the paper for the four interfaces it evaluates.

We repeated the main configuration, DeepSeek with TRACE, three more times, independently. Like most entries on the OSWorld-Verified leaderboard (94 of the 110 with a score in November 2026 report a single run; https://osworld-v1.xlang.ai/), the paper reported one run per configuration, so repetition shows how much a single run of this configuration can vary. On the three task types where the paper reports gains (A, B and C), success within three attempts varied across these runs with a standard deviation of at most 8.3 points, whereas the gains that the conclusions rest on are 29 to 75 points; on the fourth type (D), its standard deviation was 16.0 points, more than any difference among the four interfaces in Table 1, as expected since none of them lifts D clearly above the random reference. Table E3a in the Supplementary Results gives the success rates of every run, and the supplementary material gives the per-task results.

We added a control in which memory is cleared between attempts. With the same number of attempts but no memory, this control separates the benefit of remembering earlier attempts from the benefit of simply trying more often. Memory helped on the tasks whose dynamics repeat (C) and not on those whose dynamics change (D), as the paper claims: it added 41.2 points on C, about seven times the run-to-run standard deviation, but only 9.3 points on D, less than the 16.0-point variation between runs (E3; for the models with a single run, we estimate the control from their first-attempt rates).

We repeated the interface comparison on a second model. This checks that the findings do not depend on one model. The same pattern reappeared: each interface helped the kind of task it was built for, and only their combination helped on the tasks that couple perception and action (E2).

We added five external participants, for seven in total. A larger group makes the human reference more reliable, and every participant still outperformed every agent on the tasks with random dynamics (E5).

We tested a further agent interface (G1). Checking whether a stronger interface removes a claimed limit is the most direct test of that limit, and it made the boundary tighter instead of removing it.

We released the code of the experiments and the records of the main experiment. The code is in the anonymous repository (https://anonymous.4open.science/r/realtime-gui-boundaries-6C8E/), and we set it up from scratch on a fresh server and ran it. The per-task outcomes and text trajectories of all 759 task runs in virtual machines behind the paper's main table are in the repository and in the supplementary material, so every one of these numbers can be checked.

G3. [Critical] Unclear logic of the argument for the capability boundary (Reviewer Pp6d W1; Reviewer vgZz W1, Q1; Reviewer acUh W1, Q1)

We thank the reviewers for questioning the logic of our argument, which showed us that the paper invites a reversed reading. Because the paper presents the boundary (Sec. 3.3), the task design (Sec. 4) and the results (Sec. 5) separately, and the abstract says that "D tasks reveal a timing constraint", the boundary can appear to be inferred from the results. Read this way, it would need evidence failure by failure, and results on tasks designed from it would only confirm the design. The argument in fact runs the other way, in three steps. First, the boundary is derived from the turn-based loop itself (Sec. 3.3 and Appendix G, which, as Reviewer Pp6d notes, "correctly state" it), so it holds independently of the experiments. Second, the experiments serve to test the boundary, not to establish it: every task was built from the definitions of the four task types and accepted before any agent was run, and each type tests one prediction of the boundary, all of which hold (Table 1; E1 to E5). Third, we reviewed by hand 40 unexpected successes, the kind of result that could contradict the boundary, and none was a timely response to its event (E6). We apologize for this confusion and, in the camera-ready version, will state this argument in Sec. 3.3 and reword the abstract accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.