- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DevicesWorld to test whether agents can coordinate multiple independent devices to complete the same task. The benchmark contains 5,894 tasks, including 5,614 cross-device tasks, covering Android, Linux, and SmartHome. The system supports task initialization, device control, and automatic checks of task results. On the 200-task DevicesWorld-Lite subset, the best-performing model achieves a task success rate of 40%, compared with a human reference of 84%.
The authors also split cross-device tasks into single-device subtasks, test them independently, and compare the results with end-to-end execution. They find that models may still fail the original task even when they can complete all its subtasks independently.
Critical strengths
-
The benchmark covers meaningful cross-device tasks. Each device keeps its own information and state, and agents need to read information from one device and use it to act on another. The benchmark supports such tasks across phones, desktops, and smart-home environments, with task initialization and result checks. The task design also identifies the sources for the expected results, giving clear requirements and a basis for checking the results.
-
The comparison between subtasks and complete tasks helps explain failures. The authors compare the two ways of running the same original tasks, showing that completing each part on its own does not guarantee success on the full task. The appendix gives examples of choosing the wrong phone, missing required control actions, and making repeated attempts in the calendar app. These examples help readers see where agents fail to complete the task.
Critical weaknesses
-
The differences from previous work on cross-device execution could be explained more clearly. CRAB and UFO³ already study cross-device execution. The paper discusses these works and shows broader device coverage and a larger task set. However, it is still unclear which task requirements are covered more fully than in earlier work. Concrete task examples would make it easier to judge the value of the new benchmark.
-
The reasons for the gap between independent subtask execution and end-to-end execution are still not clear enough. Independent subtask tests provide clear goals, correct information from earlier steps, and the corresponding initial states. Each subtask also has its own execution budget. In the full task, agents need to organize this information themselves and complete all the required actions. The failure examples and step-count analysis are helpful, but the results still do not show whether the main problem is planning, remembering information, or detecting and correcting errors.
Questions for authors
-
Compared with earlier work on cross-device execution, such as CRAB, what task requirements does this benchmark add or cover more broadly? Please explain with concrete tasks.
-
The appendix already gives examples of information-association errors, choosing the wrong device, missing actions, and repeated attempts. Based on the existing trajectories, are these errors also common in other failed tasks? Which causes are still unclear?
4 comments
Failure patterns across evaluated systems (Q2)
Q2. How common are the reported failure patterns across failed tasks and different systems?
To examine whether the reported failure patterns recur beyond the examples in the paper, we analyzed 722 failed executions from GPT-5.5, Claude Opus 4.8, Qwen3.7-Plus, UFO³, and CRAB-MABE, each evaluated on the same 200-task DevicesWorld-Lite subset. A task may fail after several problems accumulate across execution; the table summarizes each failed execution by its primary observed failure pattern.
| Primary observed failure pattern | GPT-5.5 (n=142) | Opus 4.8 (n=125) | Qwen3.7-Plus (n=161) | UFO³ (n=149) | CRAB-MABE (n=145) | Total (n=722) |
|---|---|---|---|---|---|---|
| Information/source/field association errors | 22 (15.49%) | 14 (11.20%) | 4 (2.48%) | 27 (18.12%) | 18 (12.41%) | 85 (11.77%) |
| Wrong device-instance selection | 0 (0.00%) | 3 (2.40%) | 4 (2.48%) | 1 (0.67%) | 2 (1.38%) | 10 (1.39%) |
| Omitted required operations/information or violated constraints | 42 (29.58%) | 33 (26.40%) | 21 (13.04%) | 41 (27.52%) | 20 (13.79%) | 157 (21.75%) |
| Repeated attempts / stalled local interaction | 52 (36.62%) | 50 (40.00%) | 111 (68.94%) | 59 (39.60%) | 81 (55.86%) | 353 (48.89%) |
| Other failure patterns | 26 (18.31%) | 25 (20.00%) | 21 (13.04%) | 21 (14.09%) | 24 (16.55%) | 117 (16.20%) |
| All failed executions | 142 (100%) | 125 (100%) | 161 (100%) | 149 (100%) | 145 (100%) | 722 (100%) |
The four named patterns together account for 605/722 failed executions (83.80%). Repeated attempts or stalled local interaction are the most common primary pattern overall (353/722, 48.89%) and within each of the five systems. Omissions and constraint violations account for 157/722 (21.75%), followed by information/source/field association errors at 85/722 (11.77%). Wrong device-instance selection accounts for 10/722 (1.39%). Thus, repeated attempts, omissions, and information-association errors recur beyond the appendix examples, while wrong-device selection errors also occur but are less common.
Clarifying the causes of end-to-end failures (W2)
W2. Is the gap between independent subtask success and end-to-end execution mainly due to planning, information maintenance, or error detection and correction?
Our trajectory analysis points most strongly to difficulties in detecting stalled progress and taking effective corrective action. Information maintenance and planning are closely involved: agents need the relevant context and an updated plan to recover from an unsuccessful strategy.
1. Error detection and correction are particularly important when execution stalls. Repeated attempts or stalled interaction account for 353/722 failures (48.89%), the most frequent pattern in all systems (Q2). In the calendar case, UFO³ already has a plan that includes reading the Linux schedule, but spends all 49 device interactions on Android without obtaining the required meeting fields. It neither uses the available Linux search clue nor revises its plan (Appendix C.3). The problem is not simply the absence of an initial plan: repeated unsuccessful interactions do not lead to effective corrective action.
The climate-control case shows another failure of execution checks: the agent records controls as “applied” without issuing the commands, leaving both devices off (Appendix C.2). Together, these cases highlight the need to check progress and actual action effects, detect unfinished work, and change strategy or perform missing operations. Agent S addresses a related difficulty through a trajectory reflector that suggests alternative strategies and helps avoid repetitive actions [1].
2. Information maintenance provides the context needed for later actions and recovery. We evaluated GPT-6 with and without shared notes over three rounds on Diagnostic-60. Both conditions use 50 calls/1,800 seconds and the same history configuration. The note condition adds a shared read/write interface and prompt instructions. The agent chooses what to record and retrieve; note operations count toward the existing budget, and notes are reset between tasks.
| Configuration | Successful runs | Success rate |
|---|---|---|
| B: Standard configuration | 76/180 | 42.22% |
| N: Standard configuration with shared notes | 95/180 | 52.78% |
Adding shared notes increases the success rate from 42.22% to 52.78%, a gain of 10.56 percentage points under the same execution budget.
We also analyzed Diagnostic-60's task dependencies, finding 244 required predecessor-to-successor handoffs, including 171 cross-device handoffs (70.08%). Among the 92 subtasks receiving predecessor context, 80 (86.96%) depend on multiple predecessors. Agents must therefore retain facts, their associations with devices and objects, and the completion state of earlier operations. The notes experiment supports the importance of maintaining this information throughout end-to-end execution. Mobile-Agent-v2 similarly retains task-relevant information in a memory unit and reports lower success rates when it is removed [2].
3. Planning must organize the full task and adapt to execution feedback. The calendar case illustrates why an initial decomposition alone is insufficient: the agent must also track pending stages and revise their execution when the current approach makes no progress. Information maintenance supports these decisions, while progress checks indicate when a change is needed.
Independent subtask tests provide explicit local goals and correct predecessor information or states (Appendix D.2–D.3). Under this setup, models can focus on a local objective with its prerequisites in place, achieving 79.75%–90.30% subtask success (Table D.3). This reduces the need to organize the overall task, carry context across stages, and recover from upstream mistakes. Completing the full task end to end requires agents to maintain these conditions themselves. Failures that remain undetected or uncorrected can therefore prevent later stages from succeeding even when the individual operations are within the model's capabilities.
The clearest priority suggested by these trajectories is better progress monitoring and error recovery, supported by reliable information maintenance and feedback-driven replanning. Training or harness changes can target the difficulties identified for each agent and be assessed through end-to-end task success on DevicesWorld.
[1] Agent S: An Open Agentic Framework that Uses Computers Like a Human. ICLR, 2025.
[2] Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration. NeurIPS, 2024.
Thank you again for your constructive feedback. We hope these additional analyses and experiments address your concerns, and we welcome further discussion.
Thank You for Your Careful Review — We Welcome Further Discussion
Dear Reviewer NFVy,
Thank you again for your thoughtful review and helpful suggestions.
In our responses above, we have expanded the comparison with prior cross-device work, clarified the broader task requirements covered by DevicesWorld, and illustrated these requirements with concrete examples. We have also extended the failure analysis across models and frameworks, examined task dependencies, and added a shared-note experiment to better explain the roles of planning, information maintenance, and error recovery in end-to-end execution. These analyses also address the difference between independent subtask performance and full-task success.
As the rebuttal deadline approaches, we would be grateful if you could take a look at our responses and let us know whether they address your concerns. We would be happy to clarify any remaining points.
Warm regards,
The Authors
Task requirements and coverage across device environments (Q1, W1)
Thank you for your thoughtful comments. We appreciate your recognition of the value of executable cross-device tasks and the diagnostic comparison between subtasks and end-to-end execution. Below, we address your concerns and questions point by point.
Q1 / W1. What task requirements does DevicesWorld add or cover more broadly than existing cross-device work such as CRAB and UFO³?
We agree that CRAB [1] and UFO³ [2] already study cross-device execution and dependencies. DevicesWorld expands the scale and coverage of executable cross-device evaluation across device configurations, application scenarios, and combinations of task requirements. Across the suite, tasks involve different combinations of requirements, including integrating information from multiple sources, selecting the correct device and target object, updating existing records, preserving unaffected state, and maintaining consistency across outputs. The table below compares the task coverage of the reported benchmarks.
The full task suite covers 25 applications and 17 website categories (Table 1). Its seven environment-type combinations cover both multiple instances of the same environment type and combinations of different types, including 1,876 Android+Linux tasks and 2,995 tasks combining Android and/or Linux with SmartHome (Table B.2). These configurations bring information retrieval, application operations, file processing, and device control into a shared executable task suite.
The tasks also combine different requirements for successful execution. For example, the calendar-coordination task in Appendix B.1 requires the agent to match phone calendar entries to Linux schedule records, update only the corresponding meetings, preserve unmatched entries, and record the old and new values of every changed field. The climate-control task in Appendix C.2 combines desktop rules, current IoT device capabilities, permitted control actions, and accurate recording of their outcomes. These tasks require information association, selective updates, preservation of existing state, and consistency between actions and records in different combinations.
The guest-preparation task in Appendix D.2 illustrates how several of these requirements occur together across four device instances: one Linux desktop, two Android phones, and SmartHome. A Linux coordination note specifies that lemon should be added to the guest's snack and that the bedroom light should be set to 55%. The agent must satisfy all of the following requirements:
The native checks verify the original recipe's identity and contents, the original task's identity and completion state, and the actual lighting state. DevicesWorld evaluates different combinations of information integration, correct device and object selection, constrained updates, and consistency across multiple outputs. Its scale and breadth extend the coverage of these requirements across varied applications and both same-type and mixed-type device configurations.
[1] Tianqi Xu et al. CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents. Findings of ACL 2025.
[2] Chaoyun Zhang et al. UFO³: Weaving the Digital Agent Galaxy. arXiv:2511.11332, 2025.