acceptodds
Under review as a conference paper at ICLR 2027

CUA-Lite: An Open Platform for Computer Use Agents

Abstract

Open resources for computer use agents (CUAs) — environments, datasets, and models — remain scattered: without a standard, these resources cannot be integrated and scaled; additionally, evaluation, RL, and SFT cannot share one infrastructure. We present CUA-Lite, an open platform for developing CUAs, built around three standardized abstractions: (1) Unified environment interface, under which we integrate benchmarks across desktop, browser, and mobile, plus lightweight VM-free desktop sandboxes that replicate OSWorld's desktop at much lower cost, providing over verifiable tasks; (2) Unified supervised data format, into which we convert the major public CUA datasets, from GUI grounding and understanding to full rollouts, and record our fresh rollouts from frontier CUAs on these verifiable environments; and (3) Model harnesses shared across evaluation, RL, and SFT, one per model and implemented for model families; each harness runs its model in these computer environments, producing the rollouts that evaluation and RL consume, and renders stored supervised data into the model's own training format for SFT. We release reference scores across the integrated benchmarks, and show that RL and SFT through the same harness improve open-weight models. We release our dataset, code, and models, and the platform is open to contributions.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
computer use agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AmfMOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper aims to address an important problem of current research on CUA is not standardized in terms of model, environment, and data. The authors introduced Lite.Gym (for unified environment) and Lite.Sample (for unified trajectory format) with standardized model harness execution. The paper includes comprehensive experiments showing that their implementation results in similar performance with original ones, and the authors performed evaluation on 15 models and 12 benchmarks.

Critical strengths
  • The authors integrated several important aspects of CUA, including unifying environments and data, while standardizing model harness execution. This has important practical value to the CUA research community for easier plugin and standardized evaluation.
  • The resulting unified environment enables similar performance compared to the raw performance.
  • The evaluation is very comprehensive on 12 benchmarks and 15 models.
Critical weaknesses
  • The paper didn't demonstrate whether there is actually benefit in unifying environment and training data, despite making evaluation more standardized and easier. It would be great if the authors could show that without standardization, it is difficult to train a good model, etc.
  • Missing references: Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents. AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learning.
Questions for authors

N/A

2 comments

We thank reviewer AmfM for recognizing the practical value of unifying environments, data, and harnesses, and the breadth of our evaluation. In response to your questions:

W1: Benefit of unifying environments and training data.

While we have not demonstrated that unification is required to train a good model, it lets any model train on data from any source, whether public datasets or other models' rollouts. For example, the distillation ablations from two teachers (GPT-5.5 and Qwen3.5-27B) in Section 4.1, under different reasoning and screenshot settings across three platforms (Table G.1), would otherwise be much more difficult to run. The same format also makes it easy to combine data from many sources. We have already curated 1.26M samples from 9 public datasets and 82K verified rollouts in this format (Section 2.2), which we believe will facilitate future model training.


W2: Missing references.

Thank you for pointing out these works. We have cited Agent Data Protocol [1] and AgentOhana [2] in the related work of the updated PDF.

If these answers do not fully address your concerns, we are happy to clarify further. We will integrate the new results into the revised version.

[1] Yueqi Song, Ketan Ramaneti, Zaid Sheikh, et al. Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents. ICLR 2026.

[2] Jianguo Zhang, Tian Lan, Rithesh Murthy, et al. AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learning. arXiv:2402.15506, 2024.

Thanks for the response. I will maintain my positive score.

Reviewer_r3vUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

CUA-Lite connects computer-use agent evaluation, supervised fine-tuning, and reinforcement learning through a common environment interface, data format, and model harnesses. It integrates 12 benchmarks, harnesses for 14 model families, about 1.26 million converted samples, and more than 30,000 verifiable containerized desktop training tasks. Across 13 models, Lite.OSWorld differs from OSWorld by 2.7 percentage points on average, with a Spearman model-ranking correlation of 0.98.

Critical strengths
  • A shared interface across data, environments, evaluation, and training offers substantial practical reuse.
  • The paper measures both the resource savings of its desktop sandbox and its agreement with the original environment’s model scores.
  • SFT and RL experiments show that the platform can support training as well as reference evaluation.
Critical weaknesses
  • The reference results do not cover every task in every listed benchmark; for example, WebArena evaluation uses 241 of 812 tasks. The score table needs to be read alongside the appendix’s selection rules.
  • Models use different harnesses, history lengths, and reasoning settings, and most tasks receive one rollout. Their ranking therefore cannot be attributed entirely to model capability.
  • The authors have not fully audited training verifiers, and some tasks use model-based judging. Incorrect labels may enter SFT data or RL rewards. Resource comparisons also emphasize cold start and idle use rather than high-concurrency training throughput.
Questions for authors
  1. How often do Lite.OSWorld and OSWorld disagree at the individual-task level, and which applications or operations account for most disagreements?
  2. Are the main model rankings stable under matched history lengths, reasoning budgets, and action constraints?
  3. Could the authors audit a sample of verifier decisions and report throughput, resource use, and task failure rates under high concurrency?
1 comment

We thank reviewer r3vU for recognizing the practical value of our shared interface, the resource savings, and the training support. In response to your questions:

W1: Reference results use subsets of some benchmarks (e.g., 241 of 812 WebArena tasks).

Evaluating a WebArena subset is common because its rollouts are slow, e.g., WebArena-Lite (165 tasks) [1, 2, 3] and WebArena-Verified Hard (258 tasks) [4]. Our subset takes the first task of each of WebArena's 241 templates, so every template is covered.


W2, Q2: Different harnesses, history lengths, and reasoning settings, and one rollout per task.

Different harnesses, history lengths, and reasoning settings. We aim to compare each model under its official settings (Table F.1), not under matched settings.

One rollout per task. We decode at temperature 0 and run each task once, following the standard protocol of OSWorld and other public leaderboards. Each benchmark has hundreds of tasks, so the noise on individual tasks largely averages out in the overall success rate.


W3, Q3: Verifier audit and throughput under high concurrency.

Verifiers. Three safeguards apply: (1) Lite.OSWorld's synthesized training tasks reuse OSWorld's evaluator functions, which score the same in our container as in the VM (Table D.1), (2) each synthesized task's scripted oracle solution scores 1 (Appendix D), and (3) our desktop RL trains only on these programmatically verified tasks.

Throughput. Reviewer Xsn6 raised the same point. We add a load test in the updated PDF that compares the two runtimes under concurrent active workloads (Figure 4 in Section 2.1.2). On one Google Cloud n2-standard-128 machine (128 vCPUs, 512 GB RAM), we run both runtimes at increasing concurrency, and both replay the same pre-generated GPT-5.5 actions on each OSWorld task. Lite.OSWorld scales nearly linearly up to 128 instances, while the OSWorld VM becomes bound by CPU at 32 instances. At this point, the VMs nearly saturate the vCPUs, so their steps slow down and boots start timing out at OSWorld's 300 s limit. At peak throughput, Lite.OSWorld completes over ten times as many rollouts per hour as the OSWorld VM, while each container uses only about a quarter of the CPU and a third of the memory of a VM on average.

Table 1: OSWorld (VM) vs. Lite.OSWorld at concurrency N. Latencies are medians, and step latency excludes the 2 s pause after each action. –: every VM boot times out (300 s) from N = 64.

NRollouts/h (VM / Lite)Reset, s (VM / Lite)Step, s (VM / Lite)Peak CPU, vCPUs (VM / Lite)
144 / 5931 / 170.58 / 0.269 / 3
16603 / 1,03042 / 160.77 / 0.2689 / 30
32430 / 2,149169 / 172.56 / 0.26121 / 60
64– / 4,079– / 20– / 0.28– / 85
128– / 6,563– / 27– / 0.42– / 103

Q1: Task-level disagreement between Lite.OSWorld and OSWorld.

We run 9 models on 332 tasks on both runtimes, and 20.9% of the rollouts get a different reward on Lite.OSWorld than on OSWorld. This matches the run-to-run variance of OSWorld itself: two runs of the same models on OSWorld differ on 20.7% of rollouts. This is because CUA environments are stochastic, especially under load. For example, a page or an application may load more slowly in one run, so the same action lands on a different screen. The differences are spread across applications, from 12.1% (LibreOffice Calc) to 28.5% (GIMP).

If these answers do not fully address your concerns, we are happy to clarify further. We will integrate the new results into the revised version.

[1] Xiao Liu, Tianjie Zhang, Yu Gu, et al. VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents. ICLR 2025.

[2] Zehan Qi, Xiao Liu, Iat Long Iong, et al. WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning. ICLR 2025.

[3] Zhepei Wei, Wenlin Yao, Yao Liu, et al. WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning. EMNLP 2025.

[4] Amine El Hattami, Megh Thakkar, Nicolas Chapados, Christopher Pal. WebArena Verified: Reliable Evaluation for Web Agents. NeurIPS 2025 Workshop on Scaling Environments for Agents.

Reviewer_Xsn6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CUA-Lite provides a common environment interface, cua sft or rl data format, and per-model harness for cuas. It integrates 12 benchmarks and nine public datasets, and supports evaluation, SFT, and RL through shared components. Containerized desktops reduce measured resource use while preserving approximate model rankings relative to OSWorld. Reference evaluations and training experiments demonstrate that the infrastructure functions across platforms.

Critical strengths
  1. The container results are the most concrete contribution. Also, IMO. The accompanying comparison uses 13 models on the same tasks, with a rank correlation of 0.98. These measurements support a cheaper approximation to the original desktop environment and give users a starting point for assessing its tradeoffs.
  2. The platform supports complete end2end eval and training workflows. secs 2–4 connect benchmarks, converted data, fresh rollouts, and training across desktop, browser, and mobile.
Critical weaknesses

Beyond the "novelty" of an engineering paper, I think the engineering contribution is good, but the central unresolved question from my side is what research contribution the proposed abstractions establish beyond conventional integration, since most experiments measure model or training effects rather than benefits attributable to the platform design. and for more details:

  1. I am not convinced that the unified design establishes enough research novelty beyond ordinary integration. Section 2 describe environment wrappers, a common conversation schema, and model adapters, but do not identify a consequential difficulty that requires these particular abstractions. The experiments do not answer this question either? Table 3 4 evaluates models and evaluate SFT choices, and Figure 7 evaluates additional GRPO. These results show that the pipeline works, but the gains cannot be attributed to its shared schema or any harness. The paper needs a concrete design comparison or analysis showing what becomes possible, more efficient, or more reliable than with a straightforward combination of existing components. The container result supports one component of that case.
  2. The scaling claim lacks evidence from active workloads. Memory and CPU measurements come from an idle window, and its cold-start measurements cover individual instances. sec 4 demonstrates concurrent training, but does not report an active-workload comparison with the VM setup. As a result, the reported savings do not tell us how many useful rollouts can be completed per unit of time or hardware, or how reset failures affect training throughput. This matters because scalable data collection and RL are central motivations.

And for writing:

  1. [Minor] The intro moves too quickly from resource fragmentation to the claim that resources cannot be integrated or scaled, while Sections 1–2 repeat the three abstractions without developing their design rationale.
Questions for authors
  1. What consequential capability or improvement comes specifically from the three abstractions, and what evidence distinguishes it from conventional integration?
  2. On a representative active workload with matched action and tool budgets, how do container throughput, resource use, and paired task outcomes compare with the VM?
2 comments

We thank reviewer Xsn6 for finding the engineering contribution good and the container results concrete, and for the valuable feedback. In response to your questions:

General comment, W1, Q1: What the abstractions contribute beyond conventional integration.

As an "infrastructure, software libraries" track paper, our focus is the practical contribution of making CUA resources usable by the whole community. However, our design is not as ordinary as it may seem, for two reasons.

  • Two-way harnesses that enable reusing data across models. Straightforward integration, as in BrowserGym [1] and OpenEnv [2], is one-way. It maps a model's native format into a standardized format so the model can run in different environments. With one-way integration, trajectories from public datasets or rollout pipelines such as OSWorld's stay in the action space and context format of the harness that produced them, and a different model cannot train on them. Our harnesses also render standardized trajectories back into each model's native format, so any trajectory can train any model. For example, Section 4 trains Qwen3.5-4B on GPT-5.5 rollouts.
  • One code path for training and evaluation, across platforms. The same harness renders SFT data, runs RL rollouts, and runs evaluation, so training and inference inputs always match. To our knowledge, CUA-Lite is the first framework to unify evaluation and training across desktop, browser, and mobile (Table 3; Appendix A).

We will add this design comparison with straightforward integration to the related work of the revised paper, which we believe also addresses your minor concern on writing.


W2, Q2: Throughput, resource use, and paired outcomes under active, concurrent workloads.

We add a load test in the updated PDF that compares the two runtimes under concurrent active workloads (Figure 4 in Section 2.1.2). On one Google Cloud n2-standard-128 machine (128 vCPUs, 512 GB RAM), we run both runtimes at increasing concurrency, and both replay the same pre-generated GPT-5.5 actions on each OSWorld task. Lite.OSWorld scales nearly linearly up to 128 instances, while the OSWorld VM becomes bound by CPU at 32 instances. At this point, the VMs nearly saturate the vCPUs, so their steps slow down and boots start timing out at OSWorld's 300 s limit. At peak throughput, Lite.OSWorld completes over ten times as many rollouts per hour as the OSWorld VM, while each container uses only about a quarter of the CPU and a third of the memory of a VM on average.

Table 1: OSWorld (VM) vs. Lite.OSWorld at concurrency N. Latencies are medians, and step latency excludes the 2 s pause after each action. –: every VM boot times out (300 s) from N = 64.

NRollouts/h (VM / Lite)Reset, s (VM / Lite)Step, s (VM / Lite)Peak CPU, vCPUs (VM / Lite)
144 / 5931 / 170.58 / 0.269 / 3
16603 / 1,03042 / 160.77 / 0.2689 / 30
32430 / 2,149169 / 172.56 / 0.26121 / 60
64– / 4,079– / 20– / 0.28– / 85
128– / 6,563– / 27– / 0.42– / 103

If these answers do not fully address your concerns, we are happy to clarify further. We will integrate the new results into the revised version.

[1] Thibault Le Sellier de Chezelles, Maxime Gasse, Alexandre Lacoste, et al. The BrowserGym Ecosystem for Web Agent Research. TMLR 2025.

[2] Meta PyTorch and Hugging Face. OpenEnv: Agentic Execution Environments. 2025.

Dear reviewer Xsn6, as the discussion phase wraps up, we’d greatly appreciate it if you could share your thoughts on our rebuttal. We’re happy to address any further questions you might have.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.