- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces RACE-Bench, which categorizes real-time GUI interactions into four task types according to perception and action requirements: continuous perception, anticipatory action, reproducible real-time interaction, and stochastic real-time interaction. By combining historical-frame access with within-turn action sequences, the paper analyzes the capability limitations of turn-based agents. The experiments primarily examine which tasks benefit from each of these two interface extensions and whether retries that retain prior experience help agents complete tasks that couple perception and action, with particular attention to difficulties commonly encountered by the evaluated systems on stochastic real-time tasks.
Critical strengths
-
The research question has practical significance, and the task taxonomy helps clarify the capabilities required for real-time interaction. GUI agents need to acquire information and act promptly while their environments continue to change. The paper separately discusses continuous perception, action timing, and their coupling under reproducible and stochastic dynamics, providing a relatively clear framework for understanding the applicability of turn-based interaction. This taxonomy helps identify the capability requirements of different real-time tasks, extending the evaluation beyond overall task success rates.
-
The interface ablation design is relatively clear, and the experiments provide valuable descriptive results. The paper introduces historical-frame access and within-turn action sequences separately, then combines them to compare the effects of these interface capabilities on different tasks. The results show that historical-frame access benefits perception tasks and action sequences benefit timed-action tasks, while these extensions remain limited on stochastic real-time tasks. These observations help clarify which interaction constraints turn-based agents can compensate for and under what conditions difficulties persist.
Critical weaknesses
-
The analysis of response latency during actual tasks remains insufficient, limiting the interpretation of the core results. The paper uses response-latency constraints to explain low success rates on stochastic real-time tasks. Although it provides separate reaction-time measurements and a theoretical latency decomposition, it does not systematically report each model's end-to-end response latency during the actual evaluations or adequately distinguish the proportions of failures attributable to response timeouts and decision errors. Moreover, the environment continues running during historical-frame queries, while querying frames and executing actions require separate responses. The effects of additional tool interactions and model calls on evaluation results therefore remain unclear. Without these analyses, it is difficult to determine whether the observations arise from model decision-making ability, response speed, or their combined effects, and the evidence for explaining specific failure mechanisms from low success rates remains insufficient.
-
The lack of independent repeated evaluations leaves the stability of the main results insufficiently verified. In the main model evaluation, only one run is retained for each model/configuration on each task. Up to three retries within a run, retaining experience, assess whether the agent can use prior experience to complete the task; they cannot substitute for independent repetitions under the same settings. Chance events and run-to-run variability may affect success rates, particularly on stochastic real-time tasks. The paper states that these results are descriptive statistics. Nevertheless, without independent repeated evaluations, it remains difficult to establish whether performance differences between configurations and models are stable and whether stochastic real-time tasks consistently yield low scores.
Questions for authors
-
To what extent does actual response latency contribute to failures on stochastic real-time tasks? Could you report each model's end-to-end response latency during the actual evaluations and further distinguish failures caused by response timeouts from those caused by decision errors? Do the additional tool interactions and model calls required for historical-frame queries noticeably affect task performance? Could you compare models with similar latency or adjust task response windows to test whether the main results remain consistent and clarify the respective roles of decision-making ability and response speed?
-
Do the main experimental conclusions hold consistently under independent repeated evaluations? Could you conduct independent repeated task evaluations under the same model and configuration settings, clearly distinguishing these repetitions from retries that retain experience within a run? In particular, for stochastic real-time tasks, could you verify whether performance differences between configurations and models, as well as consistently low overall success rates, can be reliably reproduced?
8 comments
Response to Reviewer acUh [Optional] (1/4)
Although ICLR asks for brevity, we believe that every concern in your review deserves an answer, so the rest of this response goes through your review point by point, quoting each point before our answer. Please feel free to read only the points on which a concern remains, or to ask us directly instead. We thank you for the time and effort that your review took: each of your concerns is valuable to us, and we hope to resolve all of them.
W1. [Optional] Limited analysis of response latency
W1.1 Limited analysis of latency during the tasks
"The analysis of response latency during actual tasks remains insufficient, limiting the interpretation of the core results."
Every model is far slower than our tasks allow: during the tasks, a turn took 14.7 to 85.2 s at the median, depending on the model, and never less than 2.2 s, whereas the action-timing widths of our tasks are at most 1.5 s (W1.3 gives the measurements for each model). With a margin this wide, a finer analysis of latency would not change how the core results are interpreted.
W1.2 Unclear logic of the argument for the capability boundary
"The paper uses response-latency constraints to explain low success rates on stochastic real-time tasks."
See C1: the argument runs the other way, as the boundary is derived first and the tasks were designed to test it.
W1.3 Missing end-to-end latency of each model
"Although it provides separate reaction-time measurements and a theoretical latency decomposition, it does not systematically report each model's end-to-end response latency during the actual evaluations"
Thank you for this request. During the evaluations themselves, the trajectories of the main experiment give the time of every turn, from one screenshot to the next: the median turn took 14.7 to 85.2 s, depending on the model, and no turn took less than 2.2 s. As a second measure, we ran all eight models on the Human Benchmark reaction-time test (https://humanbenchmark.com/tests/reactiontime, three trials per model) in the same virtual machines and configurations as the main experiment; from the screen turning green to the click, they took 14 to 228 s on average. For each model with TRACE, in seconds (E4 gives the full tables):
| Model | Median turn during the evaluations | Reaction test |
|---|---|---|
| GPT-6 Astra | 14.7 | 14.2 |
| MiniMax-M3 | 26.2 | 17.1 |
| Claude Fable 5.1 | 37.8 | 27.7 |
| DeepSeek (turns: version of the paper; reaction test: version 1017) | 18.1 | 29.2 |
| Kimi K3 | 67.4 | 49.2 |
| GLM-5.3-Flash | 70.7 | 93.0 |
| Qwen3.8-Max | 55.3 | 112.6 |
| Gemini 3.8 Flash | 85.2 | 227.7 |
W1.4 Unclear proportions of timeouts and decision errors
"or adequately distinguish the proportions of failures attributable to response timeouts and decision errors."
See C1, whose table distinguishes the two by where the failure occurs.
W1.5 and W1.6 Unclear effect of extra tool calls for frame queries
"Moreover, the environment continues running during historical-frame queries, while querying frames and executing actions require separate responses."
"The effects of additional tool interactions and model calls on evaluation results therefore remain unclear."
Thank you for raising this. The extra responses cost the agent nothing it could otherwise use: without the tool, it would still need a full turn to see and act, which already takes far longer than our tasks allow, whereas with it, on the tasks where the tool applies, the agent no longer needs to observe in real time. The ablation in Table 1 shows this: Video adds only the frame tool to Vanilla, the plain turn-based interface, and with it DeepSeek solves 81.8% rather than 18.2% of the A tasks on the first attempt, while B, C and D change by at most one task. On a second model, GLM-5.3-Flash, adding the frame tool, to Vanilla or to Anticipate, lowers success on no task type, on the first attempt or within three attempts (E2, the interface ablation on a second model).
Response to Reviewer acUh [Optional] (2/4)
W1 (continued). [Optional] Limited analysis of response latency
W1.7 and W1.8 Unclear roles of decision-making and response speed
"Without these analyses, it is difficult to determine whether the observations arise from model decision-making ability, response speed, or their combined effects,"
"and the evidence for explaining specific failure mechanisms from low success rates remains insufficient."
Decision-making and response speed can be told apart, because the two never act together (see the table in C1). In its first row, the observations arise from response speed: the needed information appears only shortly before the deadline, so no model responds in time, whatever its decisions. In its second row, they arise from decision-making ability: the interface gives every model the information in advance, so speed no longer matters, and the models differ in what they decide. The failure mechanism is also not inferred from the low success rates. It follows from Eq. 1 and the task design.
W2. [Optional] Missing independent repeated evaluations
W2.1 Limited evidence for the soundness of the results
"The lack of independent repeated evaluations leaves the stability of the main results insufficiently verified."
See C2.
W2.2 Single retained run per configuration
"In the main model evaluation, only one run is retained for each model/configuration on each task."
This is common practice: most entries on the OSWorld-Verified leaderboard also report a single run (94 of the 110 with a score in November 2026; https://osworld-v1.xlang.ai/). To show how much a single run of the main configuration can vary, we added the three independent runs of C2.
W2.3 Retries in place of independent repetitions
"Up to three retries within a run, retaining experience, assess whether the agent can use prior experience to complete the task; they cannot substitute for independent repetitions under the same settings."
We agree, and the three new runs share no history. Their first attempts therefore form three tries without memory, which separates what memory adds from simply trying more often: within three attempts, success on C is 5.9% without memory and 47.1% with it on average, and each run with memory solves significantly more C tasks than this control, whereas memory does not solve significantly more D tasks (E3, the three new runs with this memory-cleared control).
W2.4 Unknown effect of chance and run-to-run variability
"Chance events and run-to-run variability may affect success rates, particularly on stochastic real-time tasks."
They do, and C2 measures by how much for the main configuration.
W2.5 Results limited to descriptive statistics
"The paper states that these results are descriptive statistics."
We agree that the sentence in Appendix D.1 calling our results "descriptive task averages" sets no scope. In the camera-ready version, we will rewrite it to state that each average comes from one run, and add an exact test for the comparison behind each of the paper's four conclusions, one per task type:
- Video raises success on A tasks: with DeepSeek, from 8 to 20 of the 22 A tasks within three attempts (36.4% to 90.9%; exact McNemar test over the paired tasks, p = 0.002).
- Anticipate raises success on B tasks: from 1 to 10 of the 12 B tasks within three attempts (8.3% to 83.3%; same test, p = 0.004).
- TRACE solves C tasks through retained experience rather than extra attempts: the gain from memory in W2.3 is significant in each of the three new runs (same test, p = 0.031, 0.021 and 0.016). Across the eight models of the paper, C reaches 58.8% within three attempts against 8.6% expected from three independent first attempts (1-(1-r)^3 with their pooled first-attempt rate r = 4/136, see E3).
- None of the four interfaces reliably lifts D tasks above the random reference: in the run of the paper, neither DeepSeek with any of the four interfaces nor any of the eight models with TRACE solves significantly more D tasks within three attempts than the random policy (with DeepSeek, 3, 4, 4 and 3 of the 18 tasks, against 2.7 expected; one-sided exact test of the number of tasks solved against the per-task random references of Table 10, p > 0.25); W2.7 applies the same test to the repetitions.
Beyond these tests, the gains on A, B and C are far larger than the variation between the three runs of DeepSeek with TRACE, and they reappear on a second model (see W2.6).
Response to Reviewer acUh [Critical]
Dear Reviewer acUh,
Thank you for your careful review. We are glad that you find the research question practically significant and the task taxonomy a clear framework for when turn-based interaction applies (S1), and the interface ablation clear and its results valuable (S2).
So that we discuss the same argument, we first clarify how the paper argues for its capability boundary and how this argument attributes the failures, which our writing left unclear. We then show how much the main configuration varies under independent repetition, and close with your remaining points, which are marked [Optional] so that you can read them as time allows.
C1. [Critical] Unclear logic of the argument for the capability boundary (W1, Q1)
Thank you for stating your reading so plainly, as it showed us exactly where the paper misleads. Because our abstract says that "D tasks reveal a timing constraint" and our conclusion that D "remains a turn-based boundary because execution cannot branch on new observations", the paper reads as if the timing constraint were an explanation fitted to the low success rates. What we wish to clarify is that the argument runs the other way: the boundary is derived first, from Eq. 1, and the tasks were designed to test it. Eq. 1 states that a response which depends on new information cannot arrive in time when the window from that information to the deadline is shorter than the agent's latency. The low success rates on D tasks are therefore a prediction that the experiments confirm, not the source of the boundary (G3 sets out the full argument).
The same derivation attributes each failure by where it occurs, without measuring failures one by one. In the table, A to D are the paper's four task types (continuous perception, anticipatory action, reproducible and stochastic real-time tasks), Video is the interface with historical-frame access, and Anticipate the one with within-turn action sequences; TRACE has both.
Latency is therefore sufficient to explain all observed failures on D tasks with the paper's four interfaces: the response window there is about as wide as the action-timing widths of the tasks, at most 1.5 s (Figure 2c), whereas during the evaluations no turn of any model took less than 2.2 s, and the median turn took 14.7 to 85.2 s, depending on the model (see the table in W1.3). The models can differ only in the second row, and they do, whatever their speed: GPT-6 Astra and MiniMax-M3 are among the three fastest models on both of our latency measures (W1.3), yet within three attempts they solve 82.4% and 0% of the C tasks, and the five models that are slower than both on both measures solve 52.9% to 76.5% (G7 compares the models further). G4 gives the derivation of this attribution, and we will reword the abstract and the conclusion accordingly in the camera-ready version.
C2. [Critical] Limited evidence for the soundness of the results (W2, Q2)
Thank you for asking for this; the paper is stronger with it. We repeated the main configuration, DeepSeek with TRACE, three more times on all 69 tasks, each run independent of the others (E3). The runs use DeepSeek-V4.1-Flash-1017, which on 17 October replaced the version used in the paper (see Q2.2). On A, B and C, success within three attempts varied across the runs with a standard deviation of at most 8.3 points, whereas the gains on which the conclusions rest, from Video on A, Anticipate on B and TRACE on C, are 29 to 75 points, several times this variation. D stayed low in every run: within three attempts, it averaged 20.4% with a standard deviation of 16.0 points and never exceeded 38.9%, whereas seven external human participants averaged 86.5% with a standard deviation of 8.4 points (E5).