acceptodds
Under review as a conference paper at ICLR 2027

CUA-Speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents

Abstract

Computer use agents (CUAs), which use graphical user interfaces (GUIs) to complete tasks on a computer, have recently surpassed human performance on many standard benchmarks, including on difficult long-horizon tasks. Their capabilities are undoubtedly impressive, however, a key barrier to the widespread adoption and deployment of CUAs remains their speed and cost. Progress towards faster yet capable CUAs requires reliable evaluation of their speed, but many CUA benchmarks currently face a reproducibility crisis. Benchmarks rely on complex infrastructure with variable virtual machine and container specifications that confound the evaluation of the execution speed of CUAs. Towards addressing this gap, we propose cua-speedrun, which introduces standardized infrastructure and task sets, with a focus on evaluating the speed and efficiency of CUAs. cua-speedrun uses a uniform virtual machine setup and execution pipeline, along with a common agent interface that enables single-agent implementations to operate seamlessly across different benchmarks. Across four different CUA benchmarks, we evaluate how reasoning effort, agent harnesses, and environment latency affect performance, speed, and cost. We find no single model family is optimal for all three; none of the open-weight models are on the frontier, and also, unintuitively, for some models increasing the reasoning effort can speed up task completion, while faster environment input-output can slow down overall task completion time. We believe cua-speedrun will enable structured progress towards fast, efficient CUAs, unlocking new real-world use cases and applications.

Primary area:
datasets and benchmarks
Keywords:
computer-use-agents, benchmarks, llm agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wKkJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces cua-speedrun, an evaluation infrastructure hosted on Modal that evaluates computer-use agents on task success, wall-clock time, and model cost. Evaluation runs on "representative" task subsets: 50 of 295 OSWorld, 52 of 63 OSWorld2, 26 of 143 CUA-World, and 38 of 184 MyPCBench tasks. Each subset is chosen by minimizing the energy distance between per-task vectors of agent scores on the subset and on the full benchmark.

The authors evaluate 56 configurations on OSWorld and 21 on OSWorld2, plus GPT-6 Astra alone on CUA-World and MyPCBench. From these, they report performance–time–cost Pareto frontiers.

Critical strengths
  • A carefully designed evaluation pipeline and infrastructure with valuable engineering contributions.

    • The platform separates agent running time and environment setup time. This is carefully designed in the CUA benchmark domain. But in other domains, for example, the terminalbench-2.1 built on Harbor already has the same timing design. It also profiles action-to-observation latency (Table 1), which is normally ignored in the existing CUA benchmark. So the authors tried to truly evaluate the speed of the agent component.

    • CUA-AutoDebug found real faults in widely used tooling (Appendix I.2) and some harnesses mistranslate correct model actions (Table 20).

  • The experiments are comprehensive, and details are properly documented; hence the claims and findings are well supported. Observations are directly useful to practitioners.

Critical weaknesses
  • The speed measurement is only partly standardized. Most of the time is outside the platform's control. For example, in Table 11, agent time is 75.7 s of the 89.5 s total with GPT-6 Astra (low). The Provider load, region, rate limits, time of day, and concurrency all affect that latency, and the platform controls none of them.

  • The "standard" is tied to one commercial cloud provider, Modal. Timing is controlled only for runs hosted on Modal. If user opts for a local run using the route mentioned in Appendix C.3, then its times are not comparable. Hosting costs are excluded from the reported costs (Appendix J.5) and never stated.

  • Apart from Astra (low), experiments with configurations appear to be single runs, which may alter the observations and conclusions if each configuration is tested multiple times, as the CUA tasks naturally have large variance from run to run.

  • The task vector in Eq. 1 should be a random vector, but it is treated as fixed. ziz_i is built from a single run per agent, so it is a noisy sample, yet the energy objective in Eq. 2 treats it as a fixed quantity. My understanding is that ziz_i should be a measurement intrinsic to the task itself, not a (model, harness) capability. Consequently, the rule for choosing K (Eq. 3) inherits this noise.

  • The empirical findings are mostly previously reported or expected, and the most surprising one results from the FastCUA design.

    • There are findings already reported in the literature. "The agent harness plays an important role" and "more reasoning can increase cost without improving performance" are analyzed across scaffolds and reasoning settings in [1]. [2] also showed 1) model calls dominate CUA latency, and (2) trajectories that group actions need fewer steps.

    • Some findings are naturally expected. "No single model family dominates the frontier" and "the frontier changes across benchmarks" hold for almost any Pareto analysis.

    • "Faster infra can make the agent slower" is a design artifact, not a property of the models. FastCUA returns the screenshot before the application has updated (Appendix B.3). Stale observations are the expected result. As a result, the agent might need to compensate with more steps.

[1] HOLISTIC AGENT LEADERBOARD: THE MISSING INFRASTRUCTURE FOR AI AGENT EVALUATION. [2] OSWORLD-HUMAN: BENCHMARKING THE EFFICIENCY OF COMPUTER-USE AGENTS

Questions for authors

Please address the concerns raised in the critical weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer wKkJ (1/2)

We thank the reviewer for their thoughtful review and appreciation of our carefully designed evaluation pipeline and infrastructure, our separation of agent time from environment time and profiling of action-to-observation latency, the real faults found by CUA-AutoDebug, and our comprehensive and well-documented experiments whose observations are directly useful to practitioners.

We are happy to address your questions and suggestions below:

W1. Speed measurement and provider latency

The speed measurement is only partly standardized. Most of the time is outside the platform's control. For example, in Table 11, agent time is 75.7 s of the 89.5 s total with GPT-6 Astra (low).

We kindly note that this is exactly why our infrastructure separates agent time from environment time for every task (Sec. 2.1), as the reviewer also notes in their strengths. The share of task time spent on model calls is therefore reported directly. Further, FastCUA reduces the environment's own processing time from 12.68 s to 0.34 s per task (Sec. 3.2), so nearly all of the remaining time is spent by the agent. Further, API speed is beyond anyone's control and reflects the provider's own performance, which is exactly what any user of that model experiences. We therefore standardize everything else, use each model's official provider API whenever available, and report which provider was used and when for every configuration (Appendix M.5).

The Provider load, region, rate limits, time of day, and concurrency all affect that latency, and the platform controls none of them.

Provider load, region, rate limits, and time of day are all part of the provider's performance, which is exactly what any user of that model experiences. Running tasks in parallel does not add to the measured time, since each task is timed from its own instruction until the agent terminates, and waiting for a free slot, setup, and verification all fall outside this interval (Sec. 2.1).


W2. Dependence on Modal and hosting costs

The "standard" is tied to one commercial cloud provider, Modal. Timing is controlled only for runs hosted on Modal. If user opts for a local run using the route mentioned in Appendix C.3, then its times are not comparable.

We respectfully disagree that the standard is tied to Modal. Further, motivated by this concern, we ran GPT-6 Astra (low) five times through exactly the local route in Appendix C.3, on a university Slurm cluster with KVM, with the same agent and tasks as the five runs in Appendix F. Across the two platforms, the environment time differs by only 0.3 s (2%), and the score by only 0.4 points (mean ± std over 5 runs):

PlatformScore (%)Task time (s)Agent time (s)Env. time (s)
Modal (paper)90.8 ± 1.889.5 ± 2.275.7 ± 1.913.8 ± 0.4
Slurm cluster with KVM90.4 ± 2.381.5 ± 3.868.0 ± 3.713.5 ± 0.7

The 8.0 s difference in task time comes almost entirely from agent time (7.7 s), which includes the model API calls, since the two sets of runs were made two months apart. These runs also used between 8 and 32 tasks in parallel, and the environment time of every run stays between 12.7 s and 14.5 s. We have added these results to Appendix M.2 of the revised paper.

Hosting costs are excluded from the reported costs (Appendix J.5) and never stated.

Hosting runs the same environment for every agent, so its cost only depends on task time, which we already report as its own axis. Model cost is what differs between agents, and is what a user pays to run each one. Nonetheless, we have added the hosting cost to Appendix J.5: at Modal's list prices, each environment sandbox costs about $1.00 per hour and each agent sandbox about $0.33 per hour.


W3. Single runs per configuration

Apart from Astra (low), experiments with configurations appear to be single runs, which may alter the observations and conclusions if each configuration is tested multiple times, as the CUA tasks naturally have large variance from run to run.

We respectfully disagree that the tasks in our setup have large variance from run to run. Across five runs of GPT-6 Astra (low), 48 of the 50 tasks receive the same partial score in every run, and the mean score has a standard deviation of only 1.79 points (Appendix F, Tables 11 and 12). This is in part because we remove tasks that depend on the public internet, a major source of run-to-run variance (Sec. 2.2, Appendix H). With the new runs above, GPT-6 Astra (low) now has ten runs across two platforms and two months, and its mean score changes by only 0.4 points. The FastCUA comparison also uses five runs per setting (Sec. 3.2).

AuthorsOpenReview ↗

Response to Reviewer wKkJ (2/2)

W3. Single runs per configuration (continued)

Repeating all 77 configurations several times is not feasible. Instead, we verify that the conclusions hold on new models: motivated by the reviewers' questions, we ran full reasoning sweeps for Claude Opus 5.5, GPT-6.1 Terra, and Meta Muse Wonder 1.0 on both OSWorld and OSWorld2, and the reasoning trends in Sec. 3.2 hold across all three model families (please see our response to Reviewer i8wG).


W4. Noise in the task vector

The task vector in Eq. 1 should be a random vector, but it is treated as fixed. [...] built from a single run per agent, so it is a noisy sample

In our setting, the task vector is nearly deterministic: across five runs of GPT-6 Astra (low), 48 of the 50 tasks receive the same partial score in every run (Appendix F.3). Further, the rule for choosing K (Eq. 3) requires both rank correlations to stay above 0.95 at K - 1 and K + 1, so it does not depend on a single noisy value (Sec. 2.2).

My understanding is that [the task vector] should be a measurement intrinsic to the task itself, not a (model, harness) capability.

We respectfully disagree. The goal of selection is to preserve how agents score on the full benchmark (Sec. 2.2), so the task vector is by design the pattern of agent scores on that task: two tasks are similar when agents succeed and fail on them together. This follows standard practice in benchmark subset selection, which represents tasks by model responses (Polo et al., 2024, Perlitz et al., 2024).


W5. Novelty of the findings

There are findings already reported in the literature. "The agent harness plays an important role" and "more reasoning can increase cost without improving performance" are analyzed across scaffolds and reasoning settings in [1]. [2] also showed 1) model calls dominate CUA latency, and (2) trajectories that group actions need fewer steps.

We kindly note that we cite both works (Sec. 1, Sec. 4). Our contribution is a standard that measures performance, wall-clock time, and cost together for any agent, and these findings are what it reveals at scale, across 56 OSWorld and 21 OSWorld2 configurations. Further, our results go beyond these observations (Sec. 3.2): less reasoning can make agents slower, since lower effort can take many more steps, neither harness is uniformly better, since the same model is faster through one harness at low effort and through the other at medium and high effort, and fast agents generate fewer tokens, not tokens faster.

More importantly, these findings are not a one-time result. Since CUA-Speedrun evaluates any new agent under the same standard, it enables tracking the speed, cost, and performance of agents as new models are released, and enables future analyses beyond those in this paper.

Some findings are naturally expected. "No single model family dominates the frontier" and "the frontier changes across benchmarks" hold for almost any Pareto analysis.

We respectfully disagree. Whether a single model family dominates the frontier is an ongoing empirical question that can only be answered by measurement. In fact, a single family can dominate: the OSWorld2 time frontier is formed entirely by GPT-6 Astra configurations, while on OSWorld it is not (Sec. 3.2).

"Faster infra can make the agent slower" is a design artifact, not a property of the models. FastCUA returns the screenshot before the application has updated (Appendix B.3). Stale observations are the expected result.

We respectfully disagree that this is specific to FastCUA. In any computer-use setup, the infrastructure ensures that the keyboard or mouse action is executed, but the software itself can take time to update its interface. This is true for standard CUA implementations as well, where agents already request waits for the interface to update (1.1 s per task, Table 11). FastCUA only returns the next screenshot sooner, which makes this more visible. Waiting for the software to update is therefore the agent's responsibility, just as it is for a human user. A model could simply request a short wait (e.g., 100 ms in the Save As example in Sec. 3.2), but current models do not, and instead request another screenshot 5.8 s later. Whether an agent can take advantage of a faster environment is therefore a property of the agent, and our infrastructure makes it measurable.


Thank you again for your thoughtful review. We hope our responses and the additional experiments have addressed your concerns. If you think there are still some concerns, we look forward to addressing them as well! Alternatively, if you feel all your concerns are addressed, we would appreciate it if you could reflect that in your evaluation!

Reviewer_wKkJOpenReview ↗

Official Comment by Reviewer wKkJ

Thanks for the detailed response, which addressed my concerns in W1 and W2. However, I am still concerned with W5. Therefore, I would like to keep my original score.

Reviewer_LSmgOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CUA-Speedrun, which integrates multiple computer-use benchmarks and models into a unified evaluation infrastructure to jointly assess task performance, execution time, and model inference cost. The authors select smaller evaluation subsets by minimizing the energy distance between task-score distributions derived from existing agents, and assess their representativeness through leave-one-agent-out validation. The paper also provides a low-latency interaction implementation and analyzes how reasoning effort, action batching, and agent harnesses affect performance and efficiency.

Critical strengths

S1: The problem has practical significance. Evaluating execution time and cost alongside task success addresses practical requirements for deploying computer-use agents. The unified evaluation interface and infrastructure have value as tools for the community.

S2: The paper provides substantial empirical material. The evaluation covers multiple models and benchmarks and includes subset validation, repeated evaluations for selected configurations, and analyses of reasoning effort, batching, and interaction latency.

Critical weaknesses

W1: The subset-selection objective is misaligned with its intended use for efficiency evaluation, and the existing validation does not adequately bridge this gap. The paper aims to compare task performance, speed, and cost, yet subset selection uses only existing agents’ partial scores and exact-completion indicators. Subset size is also determined by score-ranking correlations, without explicitly considering time, cost, or the joint Pareto frontier. Tasks with similar score patterns can have substantially different execution times and costs; preserving score distributions therefore does not guarantee that efficiency differences are preserved. The post hoc validation in Appendix G provides only limited support: across nine existing configurations on a filtered pool of 295 OSWorld tasks, the mean absolute error in estimated task time remains 31.4 seconds, and only two of the three speed–performance frontier configurations from the full pool are recovered. These results do not establish the reliability of efficiency comparisons between closely matched configurations, nor can they be directly generalized to all configurations in the main evaluation or to other benchmarks. Moreover, each leave-one-agent-out fold uses a different subset. An independent evaluation of the final fixed subset on new models, after the selection method and subset size have been finalized, is still missing. The validation overfitting demonstrated in Appendix L further motivates such independent testing, although it does not directly establish that the final method itself has overfit.

W2: A standardized environment does not eliminate differences across API services, and the robustness of the main efficiency rankings remains unestablished. Network conditions, server load and queuing, inference hardware, and scheduling optimizations all affect end-to-end execution time. Standardizing the desktop environment and invocation pipeline does not control these factors. Consequently, the current results characterize system performance under the conditions at the time of evaluation, but do not disentangle agent efficiency from provider-side deployment factors. The paper acknowledges these limitations, yet repeated evaluations cover only a limited set of configurations. It does not adequately characterize uncertainty in the differences between major configurations across evaluation periods or the stability of Pareto-frontier membership. Assigning every model a common rate of 100 tokens/s is only a simplified analysis with fixed trajectories, rather than an actual experiment with matched inference resources. Model inference cost likewise depends on provider pricing and excludes parts of the deployment overhead; it cannot substitute for a measure of computational efficiency. The paper therefore does not yet provide sufficient evidence to establish its framework as a reliable and generalizable standard for efficiency comparison.

W3: The contribution remains primarily engineering integration, with limited methodological novelty and generalizable insights. Integrating existing benchmarks and models, standardizing interfaces, encoding screenshots in the background, combining requests, and reusing connections all have practical value. AutoDebug’s checks of input handling and action mappings also help ensure implementation correctness. However, the paper does not sufficiently establish a distinct methodological contribution arising from the combination of these engineering components and the existing energy-distance approach. The comparisons of reasoning effort, batching, and harnesses provide useful case studies, but their conclusions remain dependent on particular models, task sets, and execution conditions. They do not yet establish a sufficiently validated, general methodology for evaluating efficiency.

Questions for authors

Q1: Please address the concerns raised in the weaknesses.

Q2: Could the numerical discrepancies between the main text and the tables be clarified? Page 8 states that Claude Opus 5 at low reasoning effort and GPT-6 Astra at low reasoning effort both score 87.6%, and that the former generates 1.5 times as many tokens as the latter. However, configurations 12 and 13 in Table 7, together with Appendix E.2, report scores of 87.6% and 90.8%, respectively, and token counts of 1,795 and 1,579—a ratio of approximately 1.14. These passages appear to refer to the same OSWorld configurations and Figure 10. Please clarify whether different runs or aggregation procedures were used, or correct any outdated values and the corresponding discussion in the main text.

4 comments
AuthorsOpenReview ↗

Response to Reviewer LSmg (1/3)

We thank the reviewer for their thoughtful review and appreciation of the practical significance of evaluating execution time and cost alongside task success, the value of our unified evaluation interface and infrastructure as tools for the community, and our substantial empirical material across multiple models and benchmarks, including subset validation, repeated evaluations, and analyses of reasoning effort, batching, and interaction latency.

We are happy to address your questions and suggestions below:

W1. Subset selection and efficiency evaluation

W1: The paper aims to compare task performance, speed, and cost, yet subset selection uses only existing agents' partial scores and exact-completion indicators.

We kindly note that our selection analysis already evaluates time on held-out models. Even though selection uses only scores, energy-based selection reduces the held-out mean task time error from 51.0 s to 31.4 s compared with difficulty-stratified random selection (Appendix G.3), and the frontier estimated from the 50-task subsets adds no configuration outside the full-set frontier (Appendix G.4, Figure 16).

Further, motivated by this concern, we ran the same leave-one-agent-out analysis for time and cost rankings. The selected subsets preserve both rankings:

Held-out rank correlationK = 49K = 50K = 51
Score (partial)0.9670.9831.000
Mean task time0.9830.9500.983
Model cost1.0000.9431.000

Cost uses the six selection models with logged per-task costs. The only swap at K = 50 is between Kimi K3 and Muse Spark, whose full-set costs differ by only 0.6% ($0.998 vs. $1.004 per task). We have added these results to Appendix M.1 of the revised paper.

the mean absolute error in estimated task time remains 31.4 seconds, and only two of the three speed–performance frontier configurations from the full pool are recovered.

An error of 31.4 s is only 8.8% of task time, while the mean task times of the nine models range from 178 s to 935 s. The one frontier configuration not recovered, Kimi K3, is itself estimated accurately, within 0.5 points and 12 s of its full-set result (85.6% vs. 85.1%, and 833 s vs. 821 s). It leaves the estimated frontier only because Claude Sonnet 5, which is about four times faster, scores 0.4 points higher on the subset, a gap well within the 2.16-point estimation error (Appendix G.3). Further, no configuration outside the full-set frontier is added (100% precision).

Moreover, each leave-one-agent-out fold uses a different subset. An independent evaluation of the final fixed subset on new models, after the selection method and subset size have been finalized, is still missing.

We respectfully disagree that this evaluation is missing. Leave-one-agent-out is exactly this independent evaluation: in each fold, the held-out model is excluded from both the task profiles and the selection, so its subset estimate is made exactly as it would be for a new model (Appendix G, Listing 1). Each fold uses a different subset precisely so that the held-out model cannot influence its own subset. Further, all 56 OSWorld and 21 OSWorld2 configurations in the paper were evaluated on the frozen subsets (Sec. 3.1).

nor can they be directly generalized to all configurations in the main evaluation or to other benchmarks.

We kindly note that the OSWorld2 and CUA-World subsets are validated separately on their own held-out models (Appendix G.2, G.3, Figure 15, Table 14).

The validation overfitting demonstrated in Appendix L further motivates such independent testing, although it does not directly establish that the final method itself has overfit.

We agree with the reviewer that Appendix L does not show that our method overfits. Unlike the methods searched in Appendix L, energy-based selection fits no weights: each model's score is the plain average of its scores on the selected tasks (Appendix G.5). The only quantity chosen with the folds is the task count K, which must also hold at K - 1 and K + 1 (Sec. 2.2).

AuthorsOpenReview ↗

Response to Reviewer LSmg (2/3)

W2. API services and robustness of efficiency rankings

W2: A standardized environment does not eliminate differences across API services, and the robustness of the main efficiency rankings remains unestablished. [...] do not disentangle agent efficiency from provider-side deployment factors.

We kindly note that API speed is beyond anyone's control and reflects the provider's own performance, which is exactly what any user of that model experiences. A user deploys a model through its provider, so we do not aim to separate the two. Instead, we standardize everything else, including the environment, task setup, and timing (Sec. 2.1, 2.3), use each model's official provider API whenever available, and report which provider was used and when for every configuration (Appendix M.5). Our timing also separates agent time from environment time for every task (Sec. 2.1), so the provider's share of task time is reported directly. Further, FastCUA reduces the environment's own processing time from 12.68 s to 0.34 s per task (Sec. 3.2), so nearly all of the remaining time is spent by the agent.

Further, to verify that the rest of our measurement does not depend on where it runs, we ran GPT-6 Astra (low) five times on a different platform, a Slurm cluster with KVM, with the same agent and tasks as the five runs in Appendix F. Across the two platforms, the environment time differs by only 0.3 s (2%), and the score by only 0.4 points (mean ± std over 5 runs):

PlatformScore (%)Task time (s)Agent time (s)Env. time (s)
Modal (paper)90.8 ± 1.889.5 ± 2.275.7 ± 1.913.8 ± 0.4
Slurm cluster with KVM90.4 ± 2.381.5 ± 3.868.0 ± 3.713.5 ± 0.7

The 8.0 s difference in task time comes almost entirely from agent time (7.7 s), which includes the model API calls, since the two sets of runs were made two months apart. We have added these results to Appendix M.2 of the revised paper.

repeated evaluations cover only a limited set of configurations. It does not adequately characterize uncertainty in the differences between major configurations across evaluation periods

With these new runs, GPT-6 Astra (low) now has ten runs across two platforms and two months, and its mean score changes by only 0.4 points. Appendix F.3 further reports task-level repeatability across repeated runs (Table 12).

Assigning every model a common rate of 100 tokens/s is only a simplified analysis with fixed trajectories, rather than an actual experiment with matched inference resources.

We agree, and this is exactly how we present it: an estimate from the same trajectories that isolates how many tokens each agent generates (Appendix E.2, Figure 10). An experiment with matched inference resources is not possible for closed models, whose serving hardware is not public.

Model inference cost likewise depends on provider pricing and excludes parts of the deployment overhead; it cannot substitute for a measure of computational efficiency.

We never claim that model cost measures computational efficiency. Model cost is what a user pays to run the agent (Appendix J.5). We report generated tokens and interaction turns separately, along with their own frontiers (Appendix D.2, D.4).

AuthorsOpenReview ↗

Response to Reviewer LSmg (3/3)

W3. Contribution and novelty

W3: The contribution remains primarily engineering integration, with limited methodological novelty and generalizable insights.

We respectfully disagree that our contribution is limited to engineering integration. CUA-Speedrun is an evaluation infrastructure and benchmark, so the infrastructure is itself a main contribution, and as the reviewer notes in S1, "the unified evaluation interface and infrastructure have value as tools for the community." A standardized infrastructure is what makes speed and cost comparable across agents in the first place (Sec. 2.1). Further, using this infrastructure, we report findings that are not visible from scores alone (Sec. 3.2): less reasoning can make agents slower, faster infrastructure can make the agent slower, and fast agents generate fewer tokens, not tokens faster. CUA-AutoDebug also found real faults in widely used tooling (Appendix I). More importantly, these findings are not a one-time result. Since CUA-Speedrun evaluates any new agent under the same standard, it enables tracking the speed, cost, and performance of agents as new models are released, and enables future analyses beyond those in this paper.

their conclusions remain dependent on particular models, task sets, and execution conditions.

We agree that the best configuration depends on the model and task set, and this is exactly why it must be measured rather than assumed. No single model family dominates the frontier, and the frontier changes across benchmarks (Sec. 3.2). CUA-Speedrun fixes the execution conditions, so every agent is measured under one standard and can be measured again as new models are released. Further, motivated by the reviewers' questions, we ran full reasoning sweeps for Claude Opus 5.5, GPT-6.1 Terra, and Meta Muse Wonder 1.0 on both OSWorld and OSWorld2, and the reasoning trends in Sec. 3.2 hold across all three model families (please see our response to Reviewer i8wG).

the existing energy-distance approach

We never claim that energy distance itself is new (Sec. 2.2). Our contribution is using it to select representative CUA task subsets, validated on held-out models against random and IRT-inspired baselines (Appendix G.3, G.5).


Q2. Numbers on page 8

Q2: Could the numerical discrepancies between the main text and the tables be clarified?

Thanks for pointing this out. The text on page 8 used the first of the five GPT-6 Astra (low) runs (87.6%, 1,203 tokens per task), while Table 7 and Appendix E.2 report the mean of the five runs (90.8%, 1,579 tokens per task, Appendix F). We have updated the text to the five-run mean: Claude Opus 5 (low) scores 3.2 points lower than GPT-6 Astra (low) and generates 14% more tokens. The conclusion is unchanged, since at a common token rate GPT-6 Astra (low) is still faster (Appendix E.2, Figure 10).


Thank you again for your thoughtful review. We hope our responses and the additional experiments have addressed your concerns. If you think there are still some concerns, we look forward to addressing them as well! Alternatively, if you feel all your concerns are addressed, we would appreciate it if you could reflect that in your evaluation!

Thank you for the detailed response, which has addressed most of my concerns. However, I remain unconvinced that the paper offers a sufficient research contribution beyond engineering integration, and will therefore maintain my original score.

Reviewer_i8wGOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces cua-speedrun, a standardized infrastructure for evaluating agent performance, speed, and cost simultaneously. The authors validate the platform on OSWorld, OSWorld 2.0, CUA-World, and MyPCBench, achieving a substantial reduction in evaluation time. Using the platform, they study the effects of reasoning effort, agent harnesses, environment latency, batch actions, and several other factors.

Critical strengths
  1. This paper addresses an gap in CUA evaluation by measuring task performance, wall-clock speed, and inference cost together. This provides a more practical view of agent efficiency and reveals trade-offs that success rate alone cannot capture.

  2. The proposed infrastructure is clear and reusable across CUA benchmarks with standardized execution, timing, and cost logging.

  3. The analyses of reasoning effort, agent harnesses, environment latency, and batch actions provide useful insights for future CUA research.

Critical weaknesses
  1. The representative subsets are mainly validated for benchmark scores and model rankings, but it is unclear whether they also preserve rankings for wall-clock time and model cost. Since speedrun’s main goal is to evaluate performance, speed, and cost jointly, this leaves the validity of its speed and cost conclusions insufficiently supported.

  2. Although the infrastructure standardizes the VM setup and measures environment and API-related latency, the reported speed and cost still depend on the selected platform and external API conditions.

  3. The platform focuses on screenshot observations and mouse/keyboard actions, but does not evaluate other observation and action spaces, such as accessibility trees or DOM-based observations. Many frontier CUAs use such structured interfaces, so it is unclear whether the reported findings generalize beyond screenshot-based agents.

Questions for authors

Several of the key reasoning-effort analyses are based on Gemini models, with only limited comparisons to GPT and Claude. Do the same trends hold for models such as GPT and Claude Opus? Is the relationship between reasoning effort, task time, success rate, and cost consistent across model families?

3 comments
AuthorsOpenReview ↗

Response to Reviewer i8wG (1/2)

We thank the reviewer for their thoughtful review and appreciation of our work in addressing a gap in CUA evaluation by measuring performance, wall-clock speed, and cost together, our clear and reusable infrastructure with standardized execution, timing, and cost logging, and the usefulness of our analyses of reasoning effort, agent harnesses, environment latency, and batch actions for future CUA research.

We are happy to address your questions and suggestions below:

W1. Do the representative subsets preserve time and cost rankings?

The representative subsets are mainly validated for benchmark scores and model rankings, but it is unclear whether they also preserve rankings for wall-clock time and model cost.

We kindly note that our selection analysis already evaluates time on held-out models. Energy-based selection reduces the held-out mean task time error from 51.0 s to 31.4 s compared with difficulty-stratified random selection, even though selection uses only scores (Appendix G.3). Further, the speed-performance frontier estimated from the 50-task subsets recovers the full-set frontier with 100% precision, and pairwise dominance agrees for 91.7% of model pairs (Appendix G.4, Figure 16).

Further, to address this concern directly, we ran the same leave-one-agent-out analysis for time and cost rankings. The selected subsets preserve both rankings:

Held-out rank correlationK = 49K = 50K = 51
Score (partial)0.9670.9831.000
Mean task time0.9830.9500.983
Model cost1.0000.9431.000

Cost uses the six selection models with logged per-task costs. The only swap at K = 50 is between Kimi K3 and Muse Spark, whose full-set costs differ by only 0.6% ($0.998 vs. $1.004 per task). We have added these results to Appendix M.1 of the revised paper.


W2. Do speed and cost depend on the platform and API conditions?

Although the infrastructure standardizes the VM setup and measures environment and API-related latency, the reported speed and cost still depend on the selected platform and external API conditions.

We respectfully disagree that the reported speed and cost depend on the selected platform. Model cost only counts the charges for model API calls and excludes environment hosting (Appendix J), so it does not depend on the platform at all. For speed, all agents run on the same standardized infrastructure (Sec. 2.3). To further verify this, we ran GPT-6 Astra (low) five times on a different platform, a Slurm cluster with KVM, with the same agent and tasks as the five runs in Appendix F. Across the two platforms, the environment time differs by only 0.3 s (2%), and the score by only 0.4 points (mean ± std over 5 runs):

PlatformScore (%)Task time (s)Agent time (s)Env. time (s)
Modal (paper)90.8 ± 1.889.5 ± 2.275.7 ± 1.913.8 ± 0.4
Slurm cluster with KVM90.4 ± 2.381.5 ± 3.868.0 ± 3.713.5 ± 0.7

The 8.0 s difference in task time comes almost entirely from agent time (7.7 s), which includes the model API calls, since the two sets of runs were made two months apart. We have added these results to Appendix M.2 of the revised paper.

API speed, on the other hand, is beyond anyone's control and reflects the provider's own performance, which is exactly what any user of that model experiences. We standardize it by using each model's official provider API whenever available, and we report which provider was used and when for every configuration (Appendix M.5).


W3. Accessibility-tree and DOM-based observations

The platform focuses on screenshot observations and mouse/keyboard actions, but does not evaluate other observation and action spaces, such as accessibility trees or DOM-based observations. Many frontier CUAs use such structured interfaces.

Screenshot observations with mouse and keyboard actions are the most common setting in CUA benchmarks such as OSWorld and OSWorld2, while accessibility-tree and DOM-based observations are less common among frontier CUAs (Anthropic, OpenAI, Google). Further, DOM is only available for web pages, while most tasks in OSWorld, OSWorld2, CUA-World, and MyPCBench are in desktop applications. Prior work also finds that agents given DOM-based annotations underperform screenshot-only agents (Aggarwal et al., 2025).

AuthorsOpenReview ↗

Response to Reviewer i8wG (2/2)

W3. Accessibility-tree and DOM-based observations (continued)

However, motivated by the reviewer's suggestion, we evaluated GPT-6 Astra (low) on OSWorld with OSWorld's own accessibility-tree observation, both in addition to and instead of screenshots. The tree is linearized and truncated exactly as in OSWorld's baseline agents. Everything else, including the agent, tasks, and infrastructure, is unchanged. Each setting is run five times (mean ± std):

ObservationScore (%)Task time (s)Env. time (s)Steps
Screenshot89.2 ± 2.664.8 ± 3.16.3 ± 0.65.8 ± 0.3
Screenshot + accessibility tree89.6 ± 1.482.4 ± 8.221.4 ± 3.65.3 ± 0.6
Accessibility tree only80.4 ± 3.0113.7 ± 17.027.7 ± 3.57.5 ± 1.2

Adding the accessibility tree to screenshots does not significantly change the score (p = 0.80), but increases task time by 27% (p = 0.006), since fetching the tree adds about 15 s of environment time per task. Using the accessibility tree alone lowers the score by 8.8 points (p = 0.03) and increases task time by 75% (p < 0.001), as the agent needs 29% more steps. The p-values are from paired t-tests over the 50 tasks. We have added these results to Appendix M.4 of the revised paper.


Q1. Do the reasoning trends hold across model families?

Several of the key reasoning-effort analyses are based on Gemini models, with only limited comparisons to GPT and Claude. Do the same trends hold for models such as GPT and Claude Opus? Is the relationship between reasoning effort, task time, success rate, and cost consistent across model families?

We kindly note that our reasoning-effort analysis is not limited to Gemini models. Section 3.2 analyzes GPT-6 Astra at four reasoning efforts on both benchmarks: on OSWorld, medium effort is within 2 points of xhigh at 29% less time and 15% less cost, whereas on OSWorld2, high effort gains 7.8 points over medium. Appendix E further reports GPT-6 Astra and GPT-5.6 Luna at four efforts each (Luna with two harnesses), and Claude Opus 5 and Claude Sonnet 5 at two efforts each.

Further, motivated by this question, we ran full reasoning sweeps for Claude Opus 5.5, GPT-6.1 Terra, and Meta Muse Wonder 1.0 on both OSWorld and OSWorld2 (score % / time per task in s / cost per task in $):

ModelEffortOSWorldOSWorld2
Claude Opus 5.5low89.6 / 57.2 / 0.2556.6 / 628 / 1.54
high93.6 / 59.5 / 0.3577.5 / 1262 / 4.63
xhigh95.6 / 88.4 / 0.5375.6 / 1851 / 7.90
max95.6 / 311.5 / 1.4385.1 / 3050 / 15.17
GPT-6.1 Terralow85.6 / 58.0 / 0.06354.9 / 488 / 0.36
medium91.6 / 53.3 / 0.05454.6 / 531 / 0.41
high91.6 / 55.8 / 0.05657.3 / 614 / 0.48
xhigh91.6 / 59.3 / 0.06966.6 / 1058 / 0.86
max91.6 / 101.4 / 0.12568.4 / 1479 / 1.51
Meta Muse Wonder 1.0minimal83.6 / 90.9 / 0.5054.5 / 1776 / 4.36
low83.6 / 141.6 / 0.6667.9 / 2189 / 5.56
medium85.6 / 188.0 / 0.7668.6 / 3097 / 6.74
high89.6 / 214.6 / 0.8966.4 / 2760 / 7.31
xhigh89.6 / 222.1 / 0.9068.6 / 3118 / 7.89
max85.5 / 241.6 / 1.0071.3 / 3466 / 9.18

The trends from the paper hold across these model families:

  1. Less reasoning can make agents slower. On OSWorld, Terra is slower at low effort than at medium effort (58.0 s vs. 53.3 s), while scoring 6 points lower.
  2. More reasoning can increase time and cost without improving performance. Terra scores 91.6% from medium to max effort on OSWorld, while time increases from 53.3 s to 101.4 s and cost more than doubles. Opus 5.5 (max) matches xhigh on OSWorld but takes 3.5x longer. Similarly, on OSWorld2, Muse Wonder 1.0 gains only 0.7 points from low to xhigh, while time and cost increase by 42%.
  3. The useful reasoning setting depends on the benchmark. Terra saturates at medium effort on OSWorld but gains 13.8 points from medium to max on OSWorld2. Similarly, Opus 5.5 gains 6.0 points from low to max on OSWorld, but 28.5 points on OSWorld2. Muse Wonder 1.0 gains 13.4 points from minimal to low on OSWorld2, but none on OSWorld.

We have added these results to Appendix M.3 of the revised paper.


Thank you again for your thoughtful review. We hope our responses and the additional experiments have addressed your concerns. If you think there are still some concerns, we look forward to addressing them as well! Alternatively, if you feel all your concerns are addressed, we would appreciate it if you could reflect that in your evaluation!

The added time/cost ranking checks and reasoning experiments across model families answer my main questions. The accessibility-tree comparison is useful too. These results give me more confidence in the platform’s value for CUA research, and I’ll raise my overall rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.