acceptodds
Under review as a conference paper at ICLR 2027

DevicesWorld: Benchmarking Cross-Device Agents in Heterogeneous Environments

Abstract

Many real-world tasks require coordination among multiple independent devices, but existing device-control benchmarks primarily focus on a single interactive environment. Large-scale executable benchmarks for evaluating such cross-device coordination remain scarce. We introduce DevicesWorld, a large-scale executable benchmark for cross-device coordination in heterogeneous environments, comprising 5,894 tasks across mobile, desktop, and IoT environments, including 5,614 cross-device tasks. DevicesWorld provides a unified environment for cross-device interaction and evaluation, enabling agents to execute tasks across multiple independent devices with automatic evaluation. We evaluate 13 agent baselines on DevicesWorld-Lite; the best achieves a Task Success Rate of 40.00%, well below the human reference of 84.00%. To further examine how these failures relate to single-device subtask execution capabilities, we select 60 cross-device tasks from DevicesWorld-Lite, decompose them into 237 single-device subtasks, and conduct independent tests using three representative models, comparing the results with end-to-end execution on the same original tasks. The three models achieve subtask success rates of 79.75%–90.30%; however, for each model, at least two-thirds of the original tasks whose subtasks all pass independent tests still fail end to end. Combined with trajectory analysis, these findings highlight task planning and decomposition, cross-device information linking and task-progress tracking, and feedback-driven replanning as directions for improving end-to-end cross-device execution. By releasing DevicesWorld, we aim to provide a unified infrastructure for future research on cross-device agents.

Primary area:
datasets and benchmarks
Keywords:
Cross-device Collaboration, Multi-Agent Collaboration, Multi-Agent Evaluation, Multi-Agent Benchmarks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NFVyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DevicesWorld to test whether agents can coordinate multiple independent devices to complete the same task. The benchmark contains 5,894 tasks, including 5,614 cross-device tasks, covering Android, Linux, and SmartHome. The system supports task initialization, device control, and automatic checks of task results. On the 200-task DevicesWorld-Lite subset, the best-performing model achieves a task success rate of 40%, compared with a human reference of 84%.

The authors also split cross-device tasks into single-device subtasks, test them independently, and compare the results with end-to-end execution. They find that models may still fail the original task even when they can complete all its subtasks independently.

Critical strengths
  1. The benchmark covers meaningful cross-device tasks. Each device keeps its own information and state, and agents need to read information from one device and use it to act on another. The benchmark supports such tasks across phones, desktops, and smart-home environments, with task initialization and result checks. The task design also identifies the sources for the expected results, giving clear requirements and a basis for checking the results.

  2. The comparison between subtasks and complete tasks helps explain failures. The authors compare the two ways of running the same original tasks, showing that completing each part on its own does not guarantee success on the full task. The appendix gives examples of choosing the wrong phone, missing required control actions, and making repeated attempts in the calendar app. These examples help readers see where agents fail to complete the task.

Critical weaknesses
  1. The differences from previous work on cross-device execution could be explained more clearly. CRAB and UFO³ already study cross-device execution. The paper discusses these works and shows broader device coverage and a larger task set. However, it is still unclear which task requirements are covered more fully than in earlier work. Concrete task examples would make it easier to judge the value of the new benchmark.

  2. The reasons for the gap between independent subtask execution and end-to-end execution are still not clear enough. Independent subtask tests provide clear goals, correct information from earlier steps, and the corresponding initial states. Each subtask also has its own execution budget. In the full task, agents need to organize this information themselves and complete all the required actions. The failure examples and step-count analysis are helpful, but the results still do not show whether the main problem is planning, remembering information, or detecting and correcting errors.

Questions for authors
  1. Compared with earlier work on cross-device execution, such as CRAB, what task requirements does this benchmark add or cover more broadly? Please explain with concrete tasks.

  2. The appendix already gives examples of information-association errors, choosing the wrong device, missing actions, and repeated attempts. Based on the existing trajectories, are these errors also common in other failed tasks? Which causes are still unclear?

4 comments
AuthorsOpenReview ↗

Task requirements and coverage across device environments (Q1, W1)

Thank you for your thoughtful comments. We appreciate your recognition of the value of executable cross-device tasks and the diagnostic comparison between subtasks and end-to-end execution. Below, we address your concerns and questions point by point.


Q1 / W1. What task requirements does DevicesWorld add or cover more broadly than existing cross-device work such as CRAB and UFO³?

We agree that CRAB [1] and UFO³ [2] already study cross-device execution and dependencies. DevicesWorld expands the scale and coverage of executable cross-device evaluation across device configurations, application scenarios, and combinations of task requirements. Across the suite, tasks involve different combinations of requirements, including integrating information from multiple sources, selecting the correct device and target object, updating existing records, preserving unaffected state, and maintaining consistency across outputs. The table below compares the task coverage of the reported benchmarks.

DimensionCRAB Benchmark-v0 [1]NebulaBench in UFO³ [2]DevicesWorld
Task pool120 tasks, including 18 cross-environment tasks55 tasks5,894 tasks, including 5,614 cross-device tasks
Environments in the benchmarkAndroid and UbuntuWindows and Ubuntu, across a five-machine setupAndroid, Linux, and IoT (SmartHome); seven environment-type combinations
Representative task coveragePhone-to-desktop information transfer and application operationsDistributed data collection, computation, system administration, and reportingCross-device information integration, application and file updates, IoT control, and coordinated outputs across devices

The full task suite covers 25 applications and 17 website categories (Table 1). Its seven environment-type combinations cover both multiple instances of the same environment type and combinations of different types, including 1,876 Android+Linux tasks and 2,995 tasks combining Android and/or Linux with SmartHome (Table B.2). These configurations bring information retrieval, application operations, file processing, and device control into a shared executable task suite.

The tasks also combine different requirements for successful execution. For example, the calendar-coordination task in Appendix B.1 requires the agent to match phone calendar entries to Linux schedule records, update only the corresponding meetings, preserve unmatched entries, and record the old and new values of every changed field. The climate-control task in Appendix C.2 combines desktop rules, current IoT device capabilities, permitted control actions, and accurate recording of their outcomes. These tasks require information association, selective updates, preservation of existing state, and consistency between actions and records in different combinations.

The guest-preparation task in Appendix D.2 illustrates how several of these requirements occur together across four device instances: one Linux desktop, two Android phones, and SmartHome. A Linux coordination note specifies that lemon should be added to the guest's snack and that the bedroom light should be set to 55%. The agent must satisfy all of the following requirements:

TargetRequired result
Existing recipe on the first phoneRename “Evening Prep Snack” to “Evening Guest Snack”; add lemon to tea and crackers; preserve the original directions and 20-minute preparation time.
Bedroom light in SmartHomeTurn the light on and set its brightness to 55%.
Existing “Bedroom lighting” task on the second phoneAfter setting the light, add a note recording the 55% setting and mark the original task complete, without creating a duplicate.

The native checks verify the original recipe's identity and contents, the original task's identity and completion state, and the actual lighting state. DevicesWorld evaluates different combinations of information integration, correct device and object selection, constrained updates, and consistency across multiple outputs. Its scale and breadth extend the coverage of these requirements across varied applications and both same-type and mixed-type device configurations.

[1] Tianqi Xu et al. CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents. Findings of ACL 2025.

[2] Chaoyun Zhang et al. UFO³: Weaving the Digital Agent Galaxy. arXiv:2511.11332, 2025.

AuthorsOpenReview ↗

Failure patterns across evaluated systems (Q2)

Q2. How common are the reported failure patterns across failed tasks and different systems?

To examine whether the reported failure patterns recur beyond the examples in the paper, we analyzed 722 failed executions from GPT-5.5, Claude Opus 4.8, Qwen3.7-Plus, UFO³, and CRAB-MABE, each evaluated on the same 200-task DevicesWorld-Lite subset. A task may fail after several problems accumulate across execution; the table summarizes each failed execution by its primary observed failure pattern.

Primary observed failure patternGPT-5.5 (n=142)Opus 4.8 (n=125)Qwen3.7-Plus (n=161)UFO³ (n=149)CRAB-MABE (n=145)Total (n=722)
Information/source/field association errors22 (15.49%)14 (11.20%)4 (2.48%)27 (18.12%)18 (12.41%)85 (11.77%)
Wrong device-instance selection0 (0.00%)3 (2.40%)4 (2.48%)1 (0.67%)2 (1.38%)10 (1.39%)
Omitted required operations/information or violated constraints42 (29.58%)33 (26.40%)21 (13.04%)41 (27.52%)20 (13.79%)157 (21.75%)
Repeated attempts / stalled local interaction52 (36.62%)50 (40.00%)111 (68.94%)59 (39.60%)81 (55.86%)353 (48.89%)
Other failure patterns26 (18.31%)25 (20.00%)21 (13.04%)21 (14.09%)24 (16.55%)117 (16.20%)
All failed executions142 (100%)125 (100%)161 (100%)149 (100%)145 (100%)722 (100%)

The four named patterns together account for 605/722 failed executions (83.80%). Repeated attempts or stalled local interaction are the most common primary pattern overall (353/722, 48.89%) and within each of the five systems. Omissions and constraint violations account for 157/722 (21.75%), followed by information/source/field association errors at 85/722 (11.77%). Wrong device-instance selection accounts for 10/722 (1.39%). Thus, repeated attempts, omissions, and information-association errors recur beyond the appendix examples, while wrong-device selection errors also occur but are less common.

AuthorsOpenReview ↗

Clarifying the causes of end-to-end failures (W2)

W2. Is the gap between independent subtask success and end-to-end execution mainly due to planning, information maintenance, or error detection and correction?

Our trajectory analysis points most strongly to difficulties in detecting stalled progress and taking effective corrective action. Information maintenance and planning are closely involved: agents need the relevant context and an updated plan to recover from an unsuccessful strategy.

1. Error detection and correction are particularly important when execution stalls. Repeated attempts or stalled interaction account for 353/722 failures (48.89%), the most frequent pattern in all systems (Q2). In the calendar case, UFO³ already has a plan that includes reading the Linux schedule, but spends all 49 device interactions on Android without obtaining the required meeting fields. It neither uses the available Linux search clue nor revises its plan (Appendix C.3). The problem is not simply the absence of an initial plan: repeated unsuccessful interactions do not lead to effective corrective action.

The climate-control case shows another failure of execution checks: the agent records controls as “applied” without issuing the commands, leaving both devices off (Appendix C.2). Together, these cases highlight the need to check progress and actual action effects, detect unfinished work, and change strategy or perform missing operations. Agent S addresses a related difficulty through a trajectory reflector that suggests alternative strategies and helps avoid repetitive actions [1].

2. Information maintenance provides the context needed for later actions and recovery. We evaluated GPT-6 with and without shared notes over three rounds on Diagnostic-60. Both conditions use 50 calls/1,800 seconds and the same history configuration. The note condition adds a shared read/write interface and prompt instructions. The agent chooses what to record and retrieve; note operations count toward the existing budget, and notes are reset between tasks.

ConfigurationSuccessful runsSuccess rate
B: Standard configuration76/18042.22%
N: Standard configuration with shared notes95/18052.78%

Adding shared notes increases the success rate from 42.22% to 52.78%, a gain of 10.56 percentage points under the same execution budget.

We also analyzed Diagnostic-60's task dependencies, finding 244 required predecessor-to-successor handoffs, including 171 cross-device handoffs (70.08%). Among the 92 subtasks receiving predecessor context, 80 (86.96%) depend on multiple predecessors. Agents must therefore retain facts, their associations with devices and objects, and the completion state of earlier operations. The notes experiment supports the importance of maintaining this information throughout end-to-end execution. Mobile-Agent-v2 similarly retains task-relevant information in a memory unit and reports lower success rates when it is removed [2].

3. Planning must organize the full task and adapt to execution feedback. The calendar case illustrates why an initial decomposition alone is insufficient: the agent must also track pending stages and revise their execution when the current approach makes no progress. Information maintenance supports these decisions, while progress checks indicate when a change is needed.

Independent subtask tests provide explicit local goals and correct predecessor information or states (Appendix D.2–D.3). Under this setup, models can focus on a local objective with its prerequisites in place, achieving 79.75%–90.30% subtask success (Table D.3). This reduces the need to organize the overall task, carry context across stages, and recover from upstream mistakes. Completing the full task end to end requires agents to maintain these conditions themselves. Failures that remain undetected or uncorrected can therefore prevent later stages from succeeding even when the individual operations are within the model's capabilities.

The clearest priority suggested by these trajectories is better progress monitoring and error recovery, supported by reliable information maintenance and feedback-driven replanning. Training or harness changes can target the difficulties identified for each agent and be assessed through end-to-end task success on DevicesWorld.

[1] Agent S: An Open Agentic Framework that Uses Computers Like a Human. ICLR, 2025.

[2] Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration. NeurIPS, 2024.


Thank you again for your constructive feedback. We hope these additional analyses and experiments address your concerns, and we welcome further discussion.

AuthorsOpenReview ↗

Thank You for Your Careful Review — We Welcome Further Discussion

Dear Reviewer NFVy,

Thank you again for your thoughtful review and helpful suggestions.

In our responses above, we have expanded the comparison with prior cross-device work, clarified the broader task requirements covered by DevicesWorld, and illustrated these requirements with concrete examples. We have also extended the failure analysis across models and frameworks, examined task dependencies, and added a shared-note experiment to better explain the roles of planning, information maintenance, and error recovery in end-to-end execution. These analyses also address the difference between independent subtask performance and full-task success.

As the rebuttal deadline approaches, we would be grateful if you could take a look at our responses and let us know whether they address your concerns. We would be happy to clarify any remaining points.

Warm regards,

The Authors

Reviewer_yoQ3OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DevicesWorld, an executable benchmark designed to evaluate agents on workflows spanning Android, Linux, and simulated IoT environments. The benchmark contains 5,894 tasks, and provides unified interaction interfaces together with automatic outcome verification. The evaluation compares 13 baseline configurations on DevicesWorld-Lite, a 200-task subset. The strongest baseline reaches a 40% success rate, below the 84% human reference.

Critical strengths

I think this is a useful benchmark. Many real-world workflows require agents to retrieve information from one device and act on it elsewhere. By supporting executable tasks across mobile, desktop, and IoT settings, DevicesWorld provides useful benchmark for evaluating capabilities other benchmarks do not capture.

Critical weaknesses

As a frontier model benchmark, the tasks are too simple. Humans can complete them in around 21 steps, and GPT-6-astra already got 40% while the human performance is 84% (not 100%). This might not provide sufficient headroom for evaluating frontier models. An improved harness might saturate the benchmark. I suggest sampling more challenging and long-horizon tasks from the large task pool.

Table 1 advertises more than 5,000 tasks, but the experiments only use 200. It is not clear whether the remaining tasks intended for training, or were they intended for evaluation but left unevaluated.

Table 1 advertises GUI, API, and CLI interaction, but the results do not distinguish between these different harnesses. It looks like the results are run on GUI harness. Can you confirm it?

Questions for authors

See weakness. Can I get a clarification for these points? I think the paper can be accepted with (1) a more challenging evaluation set, (2) a clarification on the harness. It can drop api/cli from the claim, or do a comparison of supported harness and shows api/cli is not saturated (normally coding is better than GUI for frontier models)

7 comments
AuthorsOpenReview ↗

Additional challenging tasks under an enhanced execution configuration (Q1, W1)

Thank you for your thoughtful comments. We appreciate your recognition of the value of evaluating agents across mobile, desktop, and IoT environments. Below, we address your concerns and questions point by point.


Q1 / W1. Can you evaluate more challenging tasks from the full task pool and assess the benchmark’s headroom under an improved harness?

We agree that evaluating more demanding tasks is important for assessing increasingly capable agents. Following your suggestion, we conducted a supplementary evaluation of 50 additional, more complex tasks from the full task pool, all outside the original 200-task DevicesWorld-Lite subset. All 50 tasks involve at least two independent device instances, including at least one Android device.

Building on the experimental setup used in the paper, we extended the history window from 10 to 20 interaction steps and increased the per-task budget from 50 to 100 model calls and from 1,800 to 3,600 seconds. Under this enhanced harness configuration, the evaluated agent achieved a success rate of 22.00% (11/50), with a mean execution length of 61.88 steps across all 50 runs. These results show that the additional cross-device tasks remain challenging for the evaluated agent even with a longer interaction history and a larger execution budget.

On DevicesWorld-Lite, GPT-6, the best-performing baseline, achieves a success rate of 40.00%, well below the human reference of 84.00% (Table 2). This substantial gap shows that the existing evaluation set remains challenging for the evaluated agents, with considerable room for improvement. To provide further headroom as agent capabilities improve, we will consider incorporating these 50 more challenging cross-device tasks into DevicesWorld-Lite to complement the existing evaluation set.

AuthorsOpenReview ↗

Clarification of the interaction harness (Q2, W3)

Q2 / W3. Which interaction interfaces are used in the experiments, and does the benchmark remain challenging when CLI and API actions are available?

This is an important question, and we considered the availability of different interaction modes when designing our evaluation harness. The main experiments already support GUI, CLI, and API interactions according to the device environment, rather than relying on GUI interaction alone. Specifically:

  • Android: Agents observe phone screenshots and UI-element information, and operate applications through GUI actions such as tapping, scrolling, and text entry (Appendix A.3).
  • Linux: Agents can use both GUI interaction and CLI execution within the same task. They can navigate desktop applications through GUI actions or execute shell commands and scripts for operations such as file manipulation and data processing, choosing between these modes as needed during execution (Appendix A.2.2).
  • SmartHome: Agents receive structured observations of the simulated home and interact through APIs to query device capabilities, control appliances, and create schedules or workflows (Appendix A.4).

All 11 direct-interaction models use these interfaces under the same interaction protocol. The GUI/CLI/API entry in Table 1 summarizes the interfaces available through this harness, and the direct-interaction results in Table 2 were obtained with them enabled.

We agree that API and CLI access can make supported operations more direct and efficient. CoAct-1 [1] shows how programmatic execution can replace lengthy GUI sequences for file management and data processing. Beyond Browsing [2] reports that API-based agents outperform browsing-only agents on WebArena, with further gains from combining the two interaction modes. In its Office-task evaluation, UFO² [3] reports higher success rates with GUI+API control and fewer steps on tasks solved by both GUI-only and GUI+API settings.

However, not all devices and applications expose CLI or API access to the functionality required by a task. GUI interaction is natively available in common phone and desktop environments and can provide access to application functions that lack a suitable programmatic interface, even if it requires more interaction steps. Beyond Browsing [2] documents incomplete API coverage on some websites, while UFO² [3] retains GUI interaction when a suitable API is unavailable or fails. OSWorld [4] also discusses API availability and compliance with task instructions in code-based execution. These findings support our design of providing complementary interaction modes: agents can use CLI and API actions where supported, while retaining GUI access to application functions not covered by those interfaces.

To show how these interfaces are used in practice, we analyzed the action logs of five evaluated models. The first table reports each environment’s share of device actions; Linux includes both CLI and GUI actions.

ModelAndroidLinuxSmartHome
DeepSeek V4.1 Flash84.25%12.09%3.66%
Claude Opus 4.880.92%14.97%4.11%
Claude Fable 5.179.74%16.75%3.51%
GPT-5.585.85%10.67%3.47%
GPT-684.26%11.96%3.78%
Overall83.04%13.26%3.70%

The second table reports interface usage within Linux, with Linux actions as the denominator.

ModelCLIGUI
DeepSeek V4.1 Flash83.08%16.92%
Claude Opus 4.882.00%18.00%
Claude Fable 5.170.70%29.30%
GPT-5.584.07%15.93%
GPT-678.34%21.66%
Overall79.03%20.97%

Android accounts for 83.04% of device actions overall. Within Linux, CLI accounts for 79.03% of actions, with individual model proportions ranging from 70.70% to 84.07%.

The reported best success rate of 40.00% on DevicesWorld-Lite was obtained with CLI and API actions already available. The substantial CLI usage on Linux shows that agents actively use the available programmatic capabilities, rather than relying solely on GUI interaction.

[1] Linxin Song et al. CoAct-1: Computer-using Multi-Agent System with Coding Actions. ICLR 2026.

[2] Yueqi Song et al. Beyond Browsing: API-Based Web Agents. Findings of ACL 2025.

[3] Chaoyun Zhang et al. UFO²: The Desktop AgentOS. arXiv:2504.14603, 2025.

[4] Tianbao Xie et al. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurIPS 2024.

AuthorsOpenReview ↗

The full task pool and the role of DevicesWorld-Lite (Q3, W2)

Q3 / W2. Why do the main experiments evaluate 200 of the 5,894 tasks, and how are the remaining tasks intended to be used?

Evaluating the 200-task subset alone costs approximately US$4,000 per round in our setup and requires substantial execution time. Extending the main evaluation to all 5,894 tasks would exceed our available financial and time resources. We therefore used DevicesWorld-Lite, a fixed, manually checked subset of 200 tasks, for comparisons across all 13 baseline configurations, with the same task initialization and outcome verification protocol (Section 4.1).

The subset was selected with coverage of device combinations, application scenarios, and task requirements in mind (Appendix B.3). It covers all seven environment-type combinations; cross-device tasks account for 95.0% of Lite, compared with 95.25% of the full pool (Table B.2). It also covers 23 of the 25 application categories shown in Figure B.3.

The 5,694 tasks outside DevicesWorld-Lite provide a broader pool for additional evaluation and for constructing training or validation sets. The additional 50-task evaluation reported under Q1 was drawn from this pool, and we will consider adding these 50 tasks to DevicesWorld-Lite. Researchers can define disjoint splits according to their experimental objectives, while the original 200-task DevicesWorld-Lite subset remains the fixed evaluation reference for the comparisons reported in this paper.


Thank you again for your constructive feedback. We hope these additional experiments and clarifications address your concerns and welcome further discussion.

Reviewer_zyf3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce DevicesWorld, a benchmark of executable cross-device tasks over Android phones, Linux desktops and a simulated smart home, with a unified controller and automatic checkers. On a human-checked subset, the best agent is far below human performance. A decomposition study shows that models pass most single-device subtasks, yet most tasks whose subtasks all pass still fail end to end.

Critical strengths
  • Fills a gap in large-scale executable cross-device evaluation.
  • Solid infrastructure built on AndroidWorld and OSWorld, plus a smart home environment.
Critical weaknesses
  • The cross-device challenge is not isolated. Devices share no files, clipboard or apps, and the agent moves information by reading on one device and typing on another, so a task is close to a multi-app, multi-step task with sources on different machines. There is no control that runs the same task with all sources on one device. Results are not broken down by number of devices, and the subset has very few single-device tasks.
  • The paper does not explain what is unique to cross-device tasks compared with cross-app benchmarks such as GUI Odyssey and WindowsWorld. Synced apps and transfer tools, which are common in real setups, are not covered.
  • Only the human-checked subset is evaluated. The remaining tasks are LLM-generated with automatic checks only, and their error rate is unknown.
  • Each model is run once, with no variance reported. The agent frameworks CRAB and UFO3 are tested with a single base model.
Questions for authors
  • Can you report success for the same tasks with all sources on one device versus split across devices?
  • Can you break down results by number of devices, environment combination, dependency type and task length?
  • How do top models perform with a larger budget and with full history, including past screenshots?
  • Does a simple note-taking tool for carrying information between devices change success?
6 comments
AuthorsOpenReview ↗

Cross-device challenges and single-device comparisons (Q1, W1)

Thank you for your thoughtful comments. We appreciate your recognition of DevicesWorld's contribution to executable cross-device evaluation and the quality of its infrastructure. Below, we address your concerns and questions point by point.


Q1 / W1. How do cross-device tasks differ from single-device cross-app tasks, and what happens when the same tasks are consolidated onto one device?

We agree that cross-device and single-device cross-app tasks share challenges in information integration, application navigation, and multi-step execution. DevicesWorld additionally requires agents to coordinate actions across independent device instances, each with its own environment, including its resources, application states, and available operations. The agent must associate information with the correct device, select the appropriate destination for each action, and maintain consistency across the resulting device states.

These requirements become particularly important across heterogeneous device environments. Android, Linux, and SmartHome differ in their applications, capabilities, interaction interfaces, and operational constraints. In our evaluated setting, mobile interaction uses GUI actions, desktop interaction supports GUI and CLI actions, and SmartHome interaction uses APIs. For example, the humidification task in Appendix C.1.2 requires combining an alarm from one phone, a room mapping from Linux, and configuration information from another phone, then creating a SmartHome schedule and updating the original task on the correct phone. Consolidating such a task onto one device would change the available operations and the device-specific requirements of the original task.

To directly address the requested comparison, we selected 30 tasks involving multiple devices of the same type, for which consolidation was feasible. We compared the original multi-device tasks with consolidated single-device versions while retaining their underlying information dependencies and task goals.

SettingSuccessful tasks (n/N)Success rate (SR)
Original multi-device tasks9/3030.00%
Consolidated single-device tasks19/3063.33%

Consolidation increases the success rate by 33.33 percentage points, with 10 more tasks completed. Since the underlying information dependencies are retained, this improvement provides evidence that coordinating information and actions across independent devices introduces additional difficulty. Together with the distinct capabilities and operations required across heterogeneous device environments, these results support the need for a dedicated benchmark such as DevicesWorld to evaluate cross-device coordination.

AuthorsOpenReview ↗

Performance analysis across multiple dimensions (Q2)

Q2. How does performance vary across different task dimensions?

We reported success rates (%) for all 13 evaluated configurations across device count, environment combination, dependency type, and task length. Here, n denotes the number of tasks in each group.

Device count. We count independently instantiated Android, Linux, and SmartHome environments. The ten single-device tasks are SmartHome tasks.

Model / configuration1 device (n=10)2 devices (n=95)3 devices (n=66)4 devices (n=29)
UI-Venus30.007.370.006.90
GUI-Owl50.0013.681.523.45
Qwen3.7-Plus40.0031.586.063.45
Gemini 3.1 Pro Preview80.0024.2110.6110.34
Kimi K2.660.0029.479.0910.34
Doubao Seed 2.0 Pro50.0035.799.093.45
DeepSeek V4.1 Flash40.0046.3222.7331.03
Claude Opus 4.840.0052.6322.7320.69
Claude Fable 5.180.0045.2631.8224.14
GPT-5.540.0037.8918.1820.69
GPT-630.0047.3730.3041.38
UFO³ (GPT-5.5)30.0035.7916.6710.34
CRAB-MABE (GPT-5.5)30.0034.7421.2117.24

Environment combination. A, L, and S denote Android, Linux, and SmartHome, respectively. Android-only and Linux-only groups can contain multiple devices of the same type.

Model / configurationA (n=20)L (n=18)S (n=10)A+L (n=67)A+S (n=25)L+S (n=30)A+L+S (n=30)
UI-Venus0.0011.1130.002.990.0013.333.33
GUI-Owl0.0027.7850.004.480.0020.003.33
Qwen3.7-Plus0.0066.6740.0016.420.0036.673.33
Gemini 3.1 Pro Preview5.0022.2280.0017.9112.0033.3310.00
Kimi K2.60.0055.5660.0019.4012.0033.333.33
Doubao Seed 2.0 Pro5.0061.1150.0019.408.0043.333.33
DeepSeek V4.1 Flash15.0077.7840.0032.8424.0043.3333.33
Claude Opus 4.815.0083.3340.0028.3636.0046.6736.67
Claude Fable 5.15.0072.2280.0032.8444.0053.3326.67
GPT-5.515.0077.7840.0023.8812.0036.6723.33
GPT-620.0077.7830.0046.2724.0050.0023.33
UFO³ (GPT-5.5)5.0050.0030.0020.9020.0043.3320.00
CRAB-MABE (GPT-5.5)5.0077.7830.0032.848.0030.0013.33

Dependency type. These labels can overlap because one task may contain multiple dependency patterns.

Model / configurationChain (n=73)Fan-in (n=165)Fan-out (n=39)Return (n=58)
UI-Venus2.744.850.000.00
GUI-Owl2.746.670.000.00
Qwen3.7-Plus8.2214.557.695.17
Gemini 3.1 Pro Preview10.9613.9410.2612.07
Kimi K2.610.9615.7610.265.17
Doubao Seed 2.0 Pro15.0717.5810.2612.07
DeepSeek V4.1 Flash30.1432.7323.0832.76
Claude Opus 4.834.2533.3315.3834.48
Claude Fable 5.130.1436.3633.3329.31
GPT-5.520.5524.2415.3815.52
GPT-634.2538.7925.6431.03
UFO³ (GPT-5.5)21.9222.4215.3818.97
CRAB-MABE (GPT-5.5)19.1824.2412.8215.52

Task length. We group tasks by the number of steps in successful human reference executions. The 32 tasks that failed in the reference evaluation are reported separately.

Model / configuration1–10 steps (n=12)11–20 steps (n=81)21–30 steps (n=55)>30 steps (n=20)Failed in reference evaluation (n=32)
UI-Venus33.338.640.000.003.13
GUI-Owl58.3314.810.000.003.13
Qwen3.7-Plus66.6722.2216.365.009.38
Gemini 3.1 Pro Preview75.0027.1614.555.003.13
Kimi K2.683.3323.4618.180.0012.50
Doubao Seed 2.0 Pro75.0028.4018.185.009.38
DeepSeek V4.1 Flash75.0039.5138.1820.0018.75
Claude Opus 4.866.6745.6836.3620.0018.75
Claude Fable 5.191.6743.2132.7315.0037.50
GPT-5.566.6734.5723.6415.0018.75
GPT-675.0046.9138.1815.0028.13
UFO³ (GPT-5.5)58.3328.4025.4520.009.38
CRAB-MABE (GPT-5.5)75.0029.6321.8220.0018.75

UI-Venus and GUI-Owl abbreviate UI-Venus-1.5-30B-A3B and GUI-Owl-1.5-32B-Instruct, respectively.

The analysis provides several additional findings. First, longer tasks remain challenging: averaged equally across the 13 configurations, success rates are 69.23%, 30.20%, 21.82%, and 10.77% across the four successful-reference length groups. Second, tasks spanning Android, Linux, and SmartHome remain difficult for all evaluated configurations, with success rates below 40%. Third, dependency analysis further illustrates the gap between partial completion and overall success. On fan-out tasks, GPT-6 achieves a mean partial score of 0.6620, but succeeds on only 10/39 tasks (25.64%). Together, these analyses show that longer tasks and heterogeneous environment combinations remain challenging, while progress on individual requirements often falls short of end-to-end task success, consistent with the main evaluation (Section 4.2, Lines 348–351).

AuthorsOpenReview ↗

Additional experiments on execution budget, history, and note-taking (Q3-Q4)

Q3. How does performance change with a larger execution budget and more interaction history, including past screenshots?

We first attempted to provide the full interaction history, including past screenshots. However, the resulting inputs exceeded the model's token context limit. Following testing, we therefore adopted a history window covering the most recent 20 interaction steps, including the current step, retaining textual interaction history and the corresponding screenshots. This follows a common practice in agents that operate devices: managing interaction history through limited context windows or compact summaries [1–4].

Using GPT-6, we ran each configuration three times on Diagnostic-60, giving 180 runs per configuration.

ConfigurationExecution budgetRetained historySuccess rate
B: Standard configuration50 calls; 1,800 secondsTextual history from the most recent 10 interaction steps and current device screenshots76/180 = 42.22%
E: Increased budget and extended history100 calls; 3,600 secondsMost recent 20 interaction steps, including the current step, with text and corresponding screenshots90/180 = 50.00%

The enhanced configuration improves the success rate by 7.78 percentage points. To examine the use of the larger budget, we grouped runs by whether they exceeded the original 50-step limit.

Execution lengthRunsPercentage of all runsSuccessfulUnsuccessful
≤50 steps14379.44%7469
>50 steps3720.56%1621
Total180100.00%9090

Of the 37 runs that exceeded 50 steps, 16 succeeded, showing that the additional budget can help agents complete tasks. Among the 21 unsuccessful runs exceeding 50 steps, 15 (71.43%) exhibited clear repetitive loops and exhausted the expanded budget. These results show that a larger execution budget can be beneficial, while overcoming repetitive loops remains important.


Q4. Does a simple note-taking tool for carrying information between devices change success?

We appreciate this helpful suggestion. Although shared note-taking support is not necessarily natively available across different devices and applications, such a tool could help agents retain intermediate information and carry it between devices. We therefore conducted an additional experiment by adding a lightweight shared note-taking interface to the agent harness. The agent can read and update the same note throughout a task, regardless of the device currently being operated. The interface exposes two operations:

OperationFunction
notes.readRetrieves the current note.
notes.writeWrites or replaces the note's contents.

The note is accessible throughout execution across all devices involved in the task. It remains available when earlier interactions leave the history window and is reset between tasks. We also added tool definitions and prompt instructions explaining how the agent can write and retrieve notes to preserve intermediate information. The agent decides when to use the interface and what information to record.

We kept the standard execution budget and history configuration unchanged. Note reads and writes count toward the original limit of 50 calls and 1,800 seconds. We evaluated this configuration on the same 60 tasks over three rounds, giving 180 runs.

ConfigurationSuccess rate
B: Standard configuration76/180 = 42.22%
N: Standard configuration with shared notes95/180 = 52.78%
Improvement+10.56 percentage points

DevicesWorld supports introducing such tools through the agent interface, and these results show that a simple task-local note improves cross-device task success under the same execution budget. Explicit memory operations have also been studied in prior agent research, including MemGPT [5]; here, we evaluate a lightweight implementation for maintaining information during cross-device execution.

[1] OpenCUA: Open Foundations for Computer-Use Agents. NeurIPS, 2025.

[2] Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents. COLM, 2025.

[3] AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. ICLR, 2025.

[4] CoAct-1: Computer-using Multi-agent System with Coding Actions. ICLR, 2026.

[5] MemGPT: Towards LLMs as Operating Systems. arXiv preprint, 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.