- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces cua-speedrun, an evaluation infrastructure hosted on Modal that evaluates computer-use agents on task success, wall-clock time, and model cost. Evaluation runs on "representative" task subsets: 50 of 295 OSWorld, 52 of 63 OSWorld2, 26 of 143 CUA-World, and 38 of 184 MyPCBench tasks. Each subset is chosen by minimizing the energy distance between per-task vectors of agent scores on the subset and on the full benchmark.
The authors evaluate 56 configurations on OSWorld and 21 on OSWorld2, plus GPT-6 Astra alone on CUA-World and MyPCBench. From these, they report performance–time–cost Pareto frontiers.
Critical strengths
-
A carefully designed evaluation pipeline and infrastructure with valuable engineering contributions.
-
The platform separates agent running time and environment setup time. This is carefully designed in the CUA benchmark domain. But in other domains, for example, the terminalbench-2.1 built on Harbor already has the same timing design. It also profiles action-to-observation latency (Table 1), which is normally ignored in the existing CUA benchmark. So the authors tried to truly evaluate the speed of the agent component.
-
CUA-AutoDebug found real faults in widely used tooling (Appendix I.2) and some harnesses mistranslate correct model actions (Table 20).
-
-
The experiments are comprehensive, and details are properly documented; hence the claims and findings are well supported. Observations are directly useful to practitioners.
Critical weaknesses
-
The speed measurement is only partly standardized. Most of the time is outside the platform's control. For example, in Table 11, agent time is 75.7 s of the 89.5 s total with GPT-6 Astra (low). The Provider load, region, rate limits, time of day, and concurrency all affect that latency, and the platform controls none of them.
-
The "standard" is tied to one commercial cloud provider,
Modal. Timing is controlled only for runs hosted on Modal. If user opts for a local run using the route mentioned in Appendix C.3, then its times are not comparable. Hosting costs are excluded from the reported costs (Appendix J.5) and never stated. -
Apart from Astra (low), experiments with configurations appear to be single runs, which may alter the observations and conclusions if each configuration is tested multiple times, as the CUA tasks naturally have large variance from run to run.
-
The task vector in Eq. 1 should be a random vector, but it is treated as fixed. is built from a single run per agent, so it is a noisy sample, yet the energy objective in Eq. 2 treats it as a fixed quantity. My understanding is that should be a measurement intrinsic to the task itself, not a (model, harness) capability. Consequently, the rule for choosing K (Eq. 3) inherits this noise.
-
The empirical findings are mostly previously reported or expected, and the most surprising one results from the FastCUA design.
-
There are findings already reported in the literature. "The agent harness plays an important role" and "more reasoning can increase cost without improving performance" are analyzed across scaffolds and reasoning settings in [1]. [2] also showed 1) model calls dominate CUA latency, and (2) trajectories that group actions need fewer steps.
-
Some findings are naturally expected. "No single model family dominates the frontier" and "the frontier changes across benchmarks" hold for almost any Pareto analysis.
-
"Faster infra can make the agent slower" is a design artifact, not a property of the models. FastCUA returns the screenshot before the application has updated (Appendix B.3). Stale observations are the expected result. As a result, the agent might need to compensate with more steps.
-
[1] HOLISTIC AGENT LEADERBOARD: THE MISSING INFRASTRUCTURE FOR AI AGENT EVALUATION. [2] OSWORLD-HUMAN: BENCHMARKING THE EFFICIENCY OF COMPUTER-USE AGENTS
Questions for authors
Please address the concerns raised in the critical weakness.
3 comments
Response to Reviewer wKkJ (2/2)
W3. Single runs per configuration (continued)
Repeating all 77 configurations several times is not feasible. Instead, we verify that the conclusions hold on new models: motivated by the reviewers' questions, we ran full reasoning sweeps for Claude Opus 5.5, GPT-6.1 Terra, and Meta Muse Wonder 1.0 on both OSWorld and OSWorld2, and the reasoning trends in Sec. 3.2 hold across all three model families (please see our response to Reviewer i8wG).
W4. Noise in the task vector
The task vector in Eq. 1 should be a random vector, but it is treated as fixed. [...] built from a single run per agent, so it is a noisy sample
In our setting, the task vector is nearly deterministic: across five runs of GPT-6 Astra (low), 48 of the 50 tasks receive the same partial score in every run (Appendix F.3). Further, the rule for choosing K (Eq. 3) requires both rank correlations to stay above 0.95 at K - 1 and K + 1, so it does not depend on a single noisy value (Sec. 2.2).
My understanding is that [the task vector] should be a measurement intrinsic to the task itself, not a (model, harness) capability.
We respectfully disagree. The goal of selection is to preserve how agents score on the full benchmark (Sec. 2.2), so the task vector is by design the pattern of agent scores on that task: two tasks are similar when agents succeed and fail on them together. This follows standard practice in benchmark subset selection, which represents tasks by model responses (Polo et al., 2024, Perlitz et al., 2024).
W5. Novelty of the findings
There are findings already reported in the literature. "The agent harness plays an important role" and "more reasoning can increase cost without improving performance" are analyzed across scaffolds and reasoning settings in [1]. [2] also showed 1) model calls dominate CUA latency, and (2) trajectories that group actions need fewer steps.
We kindly note that we cite both works (Sec. 1, Sec. 4). Our contribution is a standard that measures performance, wall-clock time, and cost together for any agent, and these findings are what it reveals at scale, across 56 OSWorld and 21 OSWorld2 configurations. Further, our results go beyond these observations (Sec. 3.2): less reasoning can make agents slower, since lower effort can take many more steps, neither harness is uniformly better, since the same model is faster through one harness at low effort and through the other at medium and high effort, and fast agents generate fewer tokens, not tokens faster.
More importantly, these findings are not a one-time result. Since CUA-Speedrun evaluates any new agent under the same standard, it enables tracking the speed, cost, and performance of agents as new models are released, and enables future analyses beyond those in this paper.
Some findings are naturally expected. "No single model family dominates the frontier" and "the frontier changes across benchmarks" hold for almost any Pareto analysis.
We respectfully disagree. Whether a single model family dominates the frontier is an ongoing empirical question that can only be answered by measurement. In fact, a single family can dominate: the OSWorld2 time frontier is formed entirely by GPT-6 Astra configurations, while on OSWorld it is not (Sec. 3.2).
"Faster infra can make the agent slower" is a design artifact, not a property of the models. FastCUA returns the screenshot before the application has updated (Appendix B.3). Stale observations are the expected result.
We respectfully disagree that this is specific to FastCUA. In any computer-use setup, the infrastructure ensures that the keyboard or mouse action is executed, but the software itself can take time to update its interface. This is true for standard CUA implementations as well, where agents already request waits for the interface to update (1.1 s per task, Table 11). FastCUA only returns the next screenshot sooner, which makes this more visible. Waiting for the software to update is therefore the agent's responsibility, just as it is for a human user. A model could simply request a short wait (e.g., 100 ms in the Save As example in Sec. 3.2), but current models do not, and instead request another screenshot 5.8 s later. Whether an agent can take advantage of a faster environment is therefore a property of the agent, and our infrastructure makes it measurable.
Thank you again for your thoughtful review. We hope our responses and the additional experiments have addressed your concerns. If you think there are still some concerns, we look forward to addressing them as well! Alternatively, if you feel all your concerns are addressed, we would appreciate it if you could reflect that in your evaluation!
Official Comment by Reviewer wKkJ
Thanks for the detailed response, which addressed my concerns in W1 and W2. However, I am still concerned with W5. Therefore, I would like to keep my original score.
Response to Reviewer wKkJ (1/2)
We thank the reviewer for their thoughtful review and appreciation of our carefully designed evaluation pipeline and infrastructure, our separation of agent time from environment time and profiling of action-to-observation latency, the real faults found by CUA-AutoDebug, and our comprehensive and well-documented experiments whose observations are directly useful to practitioners.
We are happy to address your questions and suggestions below:
W1. Speed measurement and provider latency
We kindly note that this is exactly why our infrastructure separates agent time from environment time for every task (Sec. 2.1), as the reviewer also notes in their strengths. The share of task time spent on model calls is therefore reported directly. Further, FastCUA reduces the environment's own processing time from 12.68 s to 0.34 s per task (Sec. 3.2), so nearly all of the remaining time is spent by the agent. Further, API speed is beyond anyone's control and reflects the provider's own performance, which is exactly what any user of that model experiences. We therefore standardize everything else, use each model's official provider API whenever available, and report which provider was used and when for every configuration (Appendix M.5).
Provider load, region, rate limits, and time of day are all part of the provider's performance, which is exactly what any user of that model experiences. Running tasks in parallel does not add to the measured time, since each task is timed from its own instruction until the agent terminates, and waiting for a free slot, setup, and verification all fall outside this interval (Sec. 2.1).
W2. Dependence on Modal and hosting costs
We respectfully disagree that the standard is tied to Modal. Further, motivated by this concern, we ran GPT-6 Astra (low) five times through exactly the local route in Appendix C.3, on a university Slurm cluster with KVM, with the same agent and tasks as the five runs in Appendix F. Across the two platforms, the environment time differs by only 0.3 s (2%), and the score by only 0.4 points (mean ± std over 5 runs):
The 8.0 s difference in task time comes almost entirely from agent time (7.7 s), which includes the model API calls, since the two sets of runs were made two months apart. These runs also used between 8 and 32 tasks in parallel, and the environment time of every run stays between 12.7 s and 14.5 s. We have added these results to Appendix M.2 of the revised paper.
Hosting runs the same environment for every agent, so its cost only depends on task time, which we already report as its own axis. Model cost is what differs between agents, and is what a user pays to run each one. Nonetheless, we have added the hosting cost to Appendix J.5: at Modal's list prices, each environment sandbox costs about $1.00 per hour and each agent sandbox about $0.33 per hour.
W3. Single runs per configuration
We respectfully disagree that the tasks in our setup have large variance from run to run. Across five runs of GPT-6 Astra (low), 48 of the 50 tasks receive the same partial score in every run, and the mean score has a standard deviation of only 1.79 points (Appendix F, Tables 11 and 12). This is in part because we remove tasks that depend on the public internet, a major source of run-to-run variance (Sec. 2.2, Appendix H). With the new runs above, GPT-6 Astra (low) now has ten runs across two platforms and two months, and its mean score changes by only 0.4 points. The FastCUA comparison also uses five runs per setting (Sec. 3.2).