acceptodds
Withdrawn from ICLR 2027

BusyWorld: Benchmarking Computer-Use Agent Planning in a Concurrent World

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Computer-use agents are moving from controlled test environments to shared realworld environments. In shared environments, multiple actors may operate on the same resources concurrently, causing task conditions to change during execution. When multiple goals compete for a limited action budget, the agent must balance their importance and urgency while considering how current choices affect future opportunities. However, existing benchmarks provide limited evaluation of such planning trade-offs under concurrency. To address this gap, we introduce BUSYWORLD, a benchmark for evaluating multi-goal planning by computer-use agents in shared concurrent environments. The two benchmark dimensions are(a) Concurrency, implemented through NPC events that can change shared state during execution, and (b) Planning trade-offs, arising from differences in goal importance, urgency, and action cost under a shared action budget. The benchmark records state changes, public interface observations, agent actions, and goal outcomes on a common timeline, enabling controlled analysis of how concurrency changes planning choices. Extensive experiments on four advanced models across 71 evaluation instances show that competition consistently lowers task success, with larger losses as more goals share the budget, and that agents often make progress toward the right goals but fail to complete them once other actors change the shared state.

Primary area:
datasets and benchmarks
Keywords:
Computer-Use Agents, Agent Evaluation, Agent Planning, Dynamic Environments, Concurrency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dnZyOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

BusyWorld evaluates computer-use agents on multi-goal planning in shared environments. There are events that change shared state during execution and goals that compete for a limited action budget. Matched static/dynamic and importance variants show that competition lowers success, and agents often reach the right goals but fail to finish them.

Critical strengths

Matched task variants keep the initial state, goals, and budget fixed while varying event activity or stated importance. This supports controlled comparisons (an important trait for benchmark paper i think). Full execution traces and metrics beyond binary success, make the dominant failure patterns easier to inspect.

Critical weaknesses

Execution bottleneck and planning trade-offs The main failure mode is reaching the cart without submitting an order, including on attribute-based selection with no NPCs. Importance edits mainly change how long agents interact, not whether orders complete. A large share of the measured gap may reflect CUA workflow competence instead of concurrency-aware multi-goal planning.

Limited scale Several key contrasts rest on small paired sets (e.g., two-goal dynamic with only a few pairs), and four-goal dynamic success falls to zero. Floor effects and sparse runs limit how well the benchmark evalaute planning strategies or models once competition is added.

Questions for authors

How do you think we can solve the problem?

Reviewer_ggEKOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a new computer use benchmark, BusyWorld, which measures the ability of agents to work within a shared environment. Agents have to balance resource constraints, and adapt according to concurrent changes made within the environment by other agents (in this case, they are scripted NPC agents, not other LLM agents).

Critical strengths
  • The setting is very interesting, and well motivated. Having multi-agent CUA evaluations are important as we increasingly move to multi-agent systems.
  • The proposed scope of the benchmark covers many topics that CUA practicioners are increasingly interested in (continual learning, concurrency, resource management).
Critical weaknesses
  • 71 instances is quite a small number. What is the bottleneck to scaling up the diversity and number of tasks? Do you have error bars that show 71 is sufficient for distinguishing the capabilities of different models?
  • There are no examples of the 71 tasks provided, and we cannot judge how feasible, realistic, or complex the tasks are.
  • The benchmark is posed as a computer use benchmark, but the paper does not describe how the operations work. In particular, most of the description of the benchmark seem to suggest it can be implemented purely in text space, with actions and tools implemented as as text based operations. Can you provide more examples of how it is a CUA evaluation? What computer apps, programs, are used? In general, the paper requires much more details of the environment setting, and visualizations of trajectories taken by the agents.
  • The only multi-agent interactions are with scripted NPC agents. It would be much more interesting and realistic if the evaluation used multiple LLM agents operating concurrently. At present, the benchmark seems more like a dynamic single-agent benchmark rather than a true multi-agent one (which requires communication and coordination).
Questions for authors
  • Can you show examples of the 71 tasks in the benchmark, and visualizations of trajectories of models on some tasks?
  • Please provide more details of how this is treated as a CUA evaluation.
  • Can you demonstrate the extensibility of this benchmark to use LLM-powered agents rather than deterministic ones?
Reviewer_iMxnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BusyWorld, a benchmark for computer-use agents in a shared shopping website. Scripted NPC shoppers buy items on a fixed schedule, so stock can run out while the agent works. Tasks have several goals that share one action budget, and the goals differ in importance and urgency. The paper tests DeepSeek-V4.1-Flash in detail and three other models once. It also tests a short text memory carried from earlier tasks to a later task. The main findings are that NPC competition lowers success, that no model finishes a four-goal task under competition, and that memory from an earlier task helps.

Critical strengths
  1. The problem is real. Agents in shared settings face other actors who change the state, and few benchmarks test this.
  2. The comparisons are clean. Matched conditions share the same NPC schedule, start state and budget. Time moves by agent actions, so model speed does not change the results.
  3. The failure analysis is detailed. Every rate comes with counts, and Tables 3 and 13 show where runs fail.
Critical weaknesses
  1. The planning trade-off part is not really measured. This is a main selling point of the paper. But under competition, both M1 and M2 get 0% success on four-goal tasks. Changing goal importance moves goal completion by only 1.4 points. So the importance change gives no usable signal. The paper also has no measure of whether the agent works on urgent or important goals first, such as the order of goals tried or an importance-weighted score. There is also no reference point, such as a human or a simple script that does urgent goals first. So we cannot tell if good planning can solve these tasks at all.
  2. The agent setup makes planning very hard. The agent only keeps three past turns (Table 5). The system prompt also says "Do not output a plan, action list, multiple actions" (Listing 1). A four-goal task takes 50 to 90 actions. With three turns of history and no plan, the agent cannot remember which goals are done. In M1 Dynamic, the two goals with no NPC competition also all fail, and the paper itself says the failures are in placing the order. So the 0% may come from this setup, not from planning under competition. Please test with a longer history, or let the agent keep a plan.
  3. Most failures come from basic website use, not from competition. In four-goal tasks, 19 of 46 failures are invalid actions or parse errors, and 24 hit the action or request limit. Attribute-based selection has no NPCs, but success is still only 8.3% and 33.3%. Gemini 3.8 Flash takes about 1.5 actions per run and never succeeds. So the benchmark mostly measures whether the agent can finish a purchase on this website, and this hides the effect of competition. Also, runs that hit the run limit are counted as failures in four-goal and selection tasks, but are removed in two-goal and memory tasks (lines 612-617). In two-goal M1 Dynamic, only 3 of 9 runs are left. The claim that two-goal tasks "restore success under competition" depends on this rule.
  4. The benchmark is small, and its release is unclear. There is one shopping website. There are only 28 base tasks (8+8+8+3+1). The count of 71 comes from adding conditions, and it counts the shared memory conditions twice (lines 606-608). The memory result comes from a single downstream task with 6 or 7 runs per condition. The three other models run once each, and all four models are Flash-level models. The paper does not say whether the website, tasks and code will be released. It also does not say where the shopping website comes from, for example whether it is built on WebArena.
Questions for authors
  1. With a longer history, or with the agent allowed to keep a plan, do four-goal tasks under competition still get 0%?
  2. Can you report whether agents try important or urgent goals first, for example the order of goals they attempt?
  3. How does a simple script that does urgent goals first perform? This would show that the tasks can be solved.
  4. Why are runs that hit the run limit removed in two-goal and memory tasks but counted as failures elsewhere? What are the two-goal and memory results if they count as failures?
  5. Will the environment, tasks and code be released? Where does the shopping website come from?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.