acceptodds
Under review as a conference paper at ICLR 2027

Multi-Agent Computer Use

Abstract

Computer use agents (CUAs) today are primarily deployed as single serial agents. This setup is suboptimal for complex long-horizon tasks that benefit from task decomposition, parallel execution, and consistent re-planning based on new information. In this paper, we argue that we should instead move towards evaluating and building *multi-agent computer use* (MACU) systems. These systems, which emphasize planning and parallel execution, alleviate many of the shortcomings of single-agent CUAs. We propose a general multi-agent setup in which a manager model decomposes computer use tasks as a directed acyclic graph (DAG) of subtasks, encoding relevant dependencies and goals for subagents. At each iteration, the manager dispatches parallel CUA subagents to carry out nodes on the ready frontier of the DAG, and continuously revises the DAG (adding, canceling, or rewriting nodes) as new findings arrive from subagents. This design treats the partially observable environment of computer use as a first class challenge: information that downstream agents may not be able to re-observe are retained and passed forward through the manager and DAG structure. We demonstrate that MACU improves over strong single-agent baselines by on four computer use benchmarks, across desktop (OSWorld) and web navigation (Online-Mind2Web, WebTailBench, Odysseys) environments, with the largest and most significant gains on multi-step and long-horizon web tasks, and exhibits more favorable test-time scaling. On Odysseys, a long-horizon web navigation benchmark, MACU solves as many tasks as the single-agent baseline under the same per-task time limit (), and matches the final single-agent success rate sooner in end-to-end wall-clock time. Our findings highlight that multi-agent coordination is a promising axis for scaling computer use agents to work productively and more effectively for longer time horizons. We release all code and interactive visualizations at *removed_for_review*.

Primary area:
foundation or frontier models, including LLMs
Keywords:
computer use agents, multimodal, multi-agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_TZEkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MACU, a multi-agent setup for computer use. A manager model splits a task into a graph of subtasks. Up to four computer use agents then work on ready subtasks at the same time, each in its own virtual machine. When a subtask finishes, the manager can change the graph: add, cancel or rewrite subtasks, pass findings to later subtasks, and pass files between machines. At the end, the manager writes the final answer and picks one machine as the final state. All workers use Qwen3.6-27B. The manager is Opus 4.6 or Qwen3.6-27B. MACU improves success rates on OSWorld (43.8 -> 48.5), Online-Mind2Web (52.2 -> 55.6), WebTailBench-v2 (20.8 -> 29.5) and Odysseys (8.5 -> 34.0)

Critical strengths
  1. The method is simple and does not depend on a specific worker model. Better workers can be plugged in directly.
  2. The gain on the long-horizon Odysseys benchmark is large. Most of it stays when the same Qwen model is used as the manager (32.0 vs. 34.0).
  3. The paper reports costs, latency and negative results honestly, for example the slower runs on Online-Mind2Web. Tables 8-10 give detailed statistics, and all prompts are included.
Critical weaknesses
  1. The single-agent baseline is limited by the time cap, so the comparison is not fair. On Odysseys, 55% of single-agent runs stop at the 3-hour limit, and 25% do so on OSWorld. Only 10 of 200 Odysseys runs reach the 200-step limit, and the average run has only about 98 actions. So the "plateau" in Fig. 3 comes from cutting runs off, not from the agent being stuck. In Fig. 6 (bottom), the single-agent curve is still going up at 180 minutes. MACU also runs four machines at once and uses 365 VM-minutes per Odysseys task, versus 145 for the single agent (Table 8). In addition, when MACU hits the time limit, the manager still writes a final answer from the partial results. This adds about 7 points at the end of the Fig. 6 curve. The paper does not say whether the single agent also gets a final answer when it times out. Please add a single-agent run with a longer time limit, or with the same total VM time. Please also give the single agent a final answer step at timeout.
  2. Some baselines are needed to show where the gain comes from: the strong manager or the multi-agent graph. On OSWorld, 42 of the 61 tasks solved only by MACU use just one worker (Table 10). In Table 13, the Opus instructions spell out almost every key press. Three baselines are missing. (a) Opus writes a detailed plan, and one Qwen agent carries it out step by step. Fig. 5 (N = 1) is close to this, but it uses only 45 tasks. (b) Several parallel single-agent runs with a judge that picks the best one, at the same compute, such as Behavior Best-of-N in Agent S3. This work is not cited. The pass@8 comparison uses only 36 tasks and a 4B worker. (c) Existing multi-agent computer use systems, such as Agent S2 or CoAct-1. Also, when the same model is both manager and worker, the gain on short tasks is near zero: +1.1 on OSWorld, and +0.2 on Online-Mind2Web in Table 1 but -1.4 in Table 7.
  3. The ablations use a small setting that may not be representative. All ablations on the planning budget, manager and worker run on 36 OSWorld tasks, mostly with a 4B worker. On this subset, the 27B worker gains 19.5 points (47.2 -> 66.7, Table 3). On the full OSWorld, it gains only 4.7. The main claims are about long web tasks, but there is no budget or follow-up ablation on Odysseys or WebTailBench. The definition of B is also unclear (see Minor 1). So the claim that ongoing replanning drives the gains is not well supported. Table 10 describes the runs but does not show cause, as the authors also note.
  4. The novelty is limited. A manager that plans a task graph and sends work to parallel workers is a known design, for example LLMCompiler, Anthropic's multi-agent research system, and Agent S2. The paper says it treats partial observability as a first-class problem. But file passing between machines is used in only 0-2.5% of tasks (Table 9). Most of the gain seems to come from retries, variants and passing findings to later subtasks. Please make clearer what is new for computer use compared with these systems.
Questions for authors
  1. When the single agent hits the time limit, does it produce a final answer? How does it do with a longer time limit?
  2. In Fig. 2, the manager missed cheaper fares ($1,169 and $1,284), but the judge still passed the "cheapest fare" rubric. How often does the Odysseys judge make this kind of mistake? Did you check some judgments by hand?
  3. How were the manager prompts developed? Were they tuned on tasks from the test benchmarks? Some prompt parts look benchmark-specific (for example tab curation, CryptPad and Hertz examples).
1 comment

We thank the reviewer for their detailed review. We are glad that the reviewer found MACU simple and independent of the underlying backbone model, and recognized its gains on long-horizon benchmarks. We address specific questions below with new experimental results, and incorporate all feedback in the updated paper.

Time limit (W1, Q1)

We experiment with forcing a final answer wherever a run lacks one, using the final-summary prompt from Jang et al. (2026). Under this setting, MACU maintains an edge over single agent, achieving a perfect success rate of 23.5% for single-agent vs. 34.0% for MACU (statistically significant with a paired t-test p-value of 0.0037).

While we agree that the time limit benefits multi-agent systems, this is a fundamental limitation of single-agent setups, and is part of our core argument that CUAs should move towards being trained and deployed as multi-agent systems. We also believe that a 3 hour time limit is generous for these tasks, which the gpt-6.1-terra experiment (see overall response) never hits.

Missing baselines (W2)

  • (a) Serial. We highlight that the proposed serial setup is a special case of MACU, obtained with N=1 or whenever the manager produces a chain DAG (e.g., Tab. 5). The manager already has the ability to produce such a DAG when warranted. For example, we observed that 86.1% of OSWorld MACU runs and 95.3% of Online-Mind2Web runs have an initial DAG with a single CUA subagent (Tab. 9). MACU is therefore strictly more general than a serial planner-executor. To provide more quantitative evidence against this baseline, we run an analysis where we executed MACU's plan serially instead of in parallel (through the DAG), and limit it to the same 3-hour limit. This serial replay (which still includes replanning and a forced final answer if missing) achieves 20.5% on Odysseys, far below the MACU result of 34.0%.
  • (b) Best-of-N. This would select among rollouts with a judge, so it is weaker than oracle pass@k, since pass@k selects perfectly. Fig. 4 of our paper compares MACU with this stronger pass@k baseline. MACU outperforms it at all action budgets (this figure counts actions from all branches of MACU, so it is a fair comparison). We now cite Agent S3.
  • (c) Multi-agent CUAs. See overall response.

We agree the gain on short tasks is limited, which fits our claim that gains concentrate on decomposable tasks, and we removed "across all benchmarks" from Sec. 4.2.

Ablation setting (W3)

We’ve updated the paper to reflect that these tasks are our dev split, which is the official OSWorld "test_small" set. We rely mostly on this smaller set for cost but note that results are generally intuitive and work well at scale. We now include results on OSWorld excluding those tasks in the appendix. After removing these tasks, MACU maintains a lead over single-agent (43.2% vs. 46.6% success rate).

Unfortunately, a full-scale budget ablation on the full OSWorld set is beyond our budget, but we conducted an analysis on the existing MACU runs to identify Odysseys success rates by the number of replans used. This gives some lower bounds: at most 0 replans achieves 12.5%, at most 5 gives 32.0%, and at most 7 gives 34.0%, which highlights that replanning is indeed helpful.

Novelty (W4)

LLMCompiler and Anthropic's multi-agent blog are focused on non-CUA domains (tool calls and text rather than GUI), and Agent S2 is serial. We updated Sec. 2 with these details. Regarding the CUA-specific capabilities, such as file passing, we highlight that this is actually the least-used of three channels (App. A.1). Other novel features we introduced in this work are also prevalently useful in achieving strong performance: VM continuation is used in 25-35% of tasks (Tab. 9), VM forks on 18-49% (Tab. 9), and forwarded findings on 11-59% (Tab. 9).

Judge (Q2)

For this specific example, the rubric checks that the "final response explicitly names which site offered the cheapest fare and states the price", which the judge applies as written (it is not able to extract the global minimum price from live sites). We use the same judge from Odysseys, which reported a high human agreement of κ=0.849\kappa = 0.849. Repeated judging agrees on ~95% of rubrics for both setups. We also manually inspected several judge results and did not identify any substantial false positives or negatives.

Prompts (Q3)

We developed the prompts on the 36-task OSWorld split that we ran ablations on. The prompt includes several generic examples created by the authors (including the notes on Google Sheets / Hertz) which are common consumer tasks and not necessarily benchmark specific.

We do not have benchmark specific prompts, and use the same setup for OSWorld, Online-Mind2Web, Odysseys, and WebTailBench. The examples do not drive the gains of MACU. MACU solves 0 of the 27 paired Odysseys tasks mentioning CryptPad or Hertz, and excluding them actually raises the success rate from 34.0% to 39.5%.

Reviewer_BjaTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MACU, a manager–worker framework for computer use. A manager LLM decomposes tasks into a DAG, while identical CUA workers execute ready subtasks in parallel on separate VMs. The manager can revise the DAG as results arrive. The system also handles VM state inheritance, snapshot-based branching, file transfer, and final VM selection.

Critical strengths
  1. Under matched action budgets, MACU performs better and uses fewer actions per solved task. These results suggest that agent organization matters for long-horizon computer use.
  2. Snapshot-based branching, VM resumption, information recovery from screenshots, and final VM selection address challenges specific to computer-use agents. Appendix E also provides a useful breakdown of the contributions of decomposition, forwarding, and restructuring.
Critical weaknesses
  1. The main comparison does not isolate the benefit of multi-agent execution. The baseline is a bare ReAct agent, while MACU adds a stronger manager that plans, inspects screenshots, retries, branches, and synthesizes answers. The results suggest that planning and supervision may explain much of the gain. The paper needs a serial planner–executor baseline with the same manager and replanning budget, as well as a comparison with an existing hierarchical CUA system.
  2. Timeout handling may inflate the Odysseys gain. About 55% of single-agent runs and 61% of MACU runs hit the three-hour limit. MACU can synthesize a final answer from partial trajectories at timeout, but it is unclear whether the baseline receives the same opportunity. The evaluation also concatenates ancestor trajectories, including failed branches, potentially giving the judge more evidence than a single-agent run.
Questions for authors
  1. Do single-agent runs receive the same opportunity to produce a final answer at timeout? Please report results with matched timeout handling and indicate how many MACU successes depend on post-timeout aggregation.
  2. Can you report a serial planner–executor baseline (Opus manager, N=1, same replanning budget) on Odysseys and WebTailBench-v2?
  3. Could you check whether judging is affected by trajectory length or failed ancestor branches?
  4. The single-agent Odysseys runs average about 98 actions over 145 minutes. Is this latency mainly due to vLLM queueing? Given the different server loads, how reliable is the reported 4.4× wall-clock comparison?
  5. Please clarify the replanning budget and make the definitions of an edit and an applied decision consistent across the paper.
  6. In Fig. 4, are MACU's actions counted across all branches or only the aggregator's ancestors?
1 comment

We thank the reviewer for their detailed review. We are glad that the reviewer found the paper clear and easy to follow, recognized that MACU performs better and uses fewer actions per solved task under matched action budgets, and appreciated that MACU addresses challenges specific to computer use agents. We are also glad that the reviewer found our ablations and analysis useful for breaking down the contributions of the various components of MACU.

We address specific questions below with new experimental results, and will incorporate all feedback in the next version of the paper.

Isolating the benefit (W1, Q2)

We highlight that the proposed serial planner-executor setup is a special case of MACU, obtained with N=1 or whenever the manager produces a chain DAG (e.g., Tab. 5). The manager already has the ability to produce such a DAG when warranted. For example, we observed that 86.1% of OSWorld MACU runs and 95.3% of Online-Mind2Web runs have an initial DAG with a single CUA subagent (Tab. 9). MACU is therefore strictly more general than a serial planner-executor.

To provide more quantitative evidence against this baseline, we run an analysis where we executed MACU's plan serially instead of in parallel (through the DAG), and limit it to the same 3-hour limit. This serial replay can be seen as a test of the serial planner-executor setup. We find that this achieves 20.5% on Odysseys, far below our result with MACU of 34.0%.

We refer the reviewer to our overall response for comparison against existing hierarchical systems.

Timeouts (W2, Q1)

We experiment with forcing a final answer wherever a run lacks one, in both setups, using the final-summary prompt from Jang et al. (2026). Under this setting, MACU maintains an edge over the single agent baselines, achieving a perfect success rate of 23.5% for single-agent vs. 34.0% for MACU (which is statistically significant with a two-sided paired t-test p-value of 0.0037).

Judge sensitivity (Q3)

We performed another check and verified that length or failed branches do not inflate success:

  • Both setups included failed attempts in judging, with MACU excluding cancelled and pruned branches based on manager decisions (as described in App. B).
  • Success rate decreases as trajectory length increases (MACU + Opus gets 42% up to 100 steps, 21% for above 200).
  • Runs with a failed ancestor succeed less often (15.1% vs 66.2%).

On accuracy, we use the same official judge from the Odysseys (Jang et al., 2026) paper, and they report high human agreement scores of κ=0.849\kappa = 0.849 (perfect) and 0.7880.788 (averaged), with the same judge that we use. We also manually inspected several judge results and did not identify any substantial false positives or negatives w.r.t. rubric scores.

Latency (Q4)

Both setups ran under comparable vLLM load: 4 parallel servers throughout each experiment, with a median of about 6 active agents per server (6.0 for single-agent, 6.3 for MACU) and a similar mean time per vllm call (95s vs 86s). Adjusting for that difference would slightly lower 4.4×4.4\times to an estimate 3.6−4.0×3.6-4.0\times which we will note in the updated appendix.

Budget (Q5)

One applied replan decision (i.e., DAG change) costs one unit of budget. We have updated the paper throughout accordingly.

Fig. 4 (Q6)

It counts actions across all branches for MACU and all attempts for pass@8.

Reviewer_DFgbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a multi-agent system called MACU for computer-use tasks. MACU uses a manager to plan, decompose, and iterate over a DAG, and operates on a graph of subagents. Multiple subagents execute in isolated VMs concurrently to reduce the execution time. Results on 4 benchmarks demonstrate that MACU achieves higher SR than single-agent baselines. The paper also provides ablation studies and analyses of its key design choices.

However, MACU does not provide a general mechanism for merging application states across VMs, limiting its effectiveness when parallel subtasks independently modify the GUI environment. The paper also lacks comparisons with stronger existing CUA frameworks, and several experimental details and reported results need clarification.

Critical strengths
  1. The main contribution of this work is to apply multi-agent parallel execution and DAG-based planning to stateful GUI environments. Although similar ideas have been explored in prior work such as LLMCompiler and Flow, these methods primarily operate on tool calls, text, and code. GUI environments are more challenging; GUI state is not easy to share and integrate. This work makes a valuable exploration of this direction.
  2. MACU improves the SR of the single-agent baseline on 4 benchmarks, with large gains on long-horizon tasks. Comprehensive ablation studies and analyses support these gains. The appendix provides sufficient details.
Critical weaknesses
  1. Subagents operating in isolated VMs introduce substantial resource overhead. More importantly, MACU can't automatically merge GUI states across different VMs. On OSWorld, only one VM is used to evaluate, potentially ignoring the work finished by other subagents. On other benchmarks, the evaluator aggregates trajectories from multiple VMs, but this doesn't guarantee that the results can coexist in a single environment. Also, it's very difficult to apply to real-world scenarios. These limitations may answer why current CUAs are primarily deployed as serial agents.
  2. This paper doesn't compare to existing strong baselines. According to the OSWorld leaderboard, claude opus 4.6 achieves 72.7% SR with Anthropic's official implementation while VLAA-GUI+claude opus 4.6 achieves 77.45% SR, both substantially higher than MACU's 48.5% with the same LLM backbone, although the qwen3.6-27b worker may drag down the performance.
  3. The definition of the replanning budget (B) is inconsistent. Lines 216-217 state that a single replan can contain multiple edits, with each edit consuming one unit of the budget. However, Alg.1 (line 15) and Appendix A state that each replan consumes one unit.
  4. OSWorld has an updated version called OSWorld-verified which fixes some issues in the original version, and it also accepts evaluation on a 361-task subset excluding 8 google drive tasks. Not sure which version the authors used. This can potentially explain why the authors can't reproduce the result of the official qwen3.6-27b CUA (63.9%).
  5. The ablation studies are conducted only on an OSWorld subset (36 tasks), with the worker changed from qwen3.6-27b to qwen3.5-4b; the same applies to Odysseys (45 tasks). The ablation results in Tab.2 differ substantially from those on the full benchmark, making it unclear whether the ablation findings generalize to stronger workers and larger task sets.
  6. The 60.4% SR in Fig.5 is questionable. The Odysseys subset contains 45 tasks and the paper defines SR as the percentage of tasks satisfying all rubrics. Under this definition, 27/45=60.0% and 28/45=62.2%, neither produces 60.4%.
  7. Tab. 7 reports confidence intervals and p-values for all benchmarks except Odysseys, why?
  8. Several references contain incorrect metadata, for example, ReAct is cited as ICLR 2022, but it was actually 2023; MetaGPT is cited as ICLR 2023, but it was actually 2024; PagedAttention is listed as SOPS 2023, but it should be SOSP. There are more.
  9. The paper has noticeable blank spaces on page 13 and pages 26-29.
Questions for authors
  1. Can the authors compare MACU against existing stronger baselines?
  2. Does one unit consume one edit or one replan decision?
  3. Which version and task subset of OSWorld were used?
  4. Can the authors clarify the result in Fig.5 (60.4%)?
  5. Can the authors provide the confidence interval and p-value for Odysseys?
4 comments

We thank the reviewer for their detailed suggestions. We are glad that they recognized the value of our work in exploring multi-agent execution in computer-use and GUI-based environments, and highlighted the challenges that we addressed in this work. We also appreciate that the reviewer highlighted the comprehensiveness of our paper and experiments, and recognized that MACU achieves large gains on long-horizon tasks.

State merging (W1)

Indeed, we do not claim that MACU merges GUI states (see App. F for more discussion). The manager consolidates results through a mix of VM inheritance and transfer of findings and files between subagents by the manager. Despite not doing exact GUI state merging, our results show that MACU is still effective in improving computer use performance. We attribute this to several reasons. Many real world CUA tasks involve information gathering or discovery, and the primary deliverable in these cases is the final response, so states from different VMs do not need to be merged (we disclose this in App. B and F, and will discuss this more given more space in a future version of the paper). Many tasks also can be parallelized, enabling faster completion times and allowing the system to trade off inference compute for stronger results. For these reasons, we believe that multi-agent computer use is still a meaningful direction and axis of scaling worth investing into.

Stronger baselines (W2, Q1)

The cited systems which score 72.7% and 77.45% use Opus 4.6 to carry out all actions, while MACU only uses Opus to carry out the planning steps, so this is not an apples-to-apples comparison, as Opus 4.6 is significantly more expensive than the Qwen3.6-27B subagent we use. VLAA-GUI also adds specialized verifier, loop-breaker, search, coding and grounding modules, so the reasoning in the overall response applies.

Unfortunately, running GPT-5.4 or Opus 4.6 as subagents is prohibitively expensive for us (we estimate ~\13pertaskaveragedacrossbenchmarks,i.e., 13 per task averaged across benchmarks, i.e., ~\\11K for this experiment). To address the reviewer’s concern however, we run MACU with gpt-6.1-terra with xhigh reasoning (a cheaper but close-to-frontier model) as both the subagent and the manager on the 109 hard Odysseys tasks, as these are the ones that latest models struggle most on (we find that gpt-6.1-terra achieves close to perfect on easy/medium difficulty tasks). We find that MACU's gains carry over to this stronger model: MACU achieves a perfect success rate of 74.3%, and a single-agent gpt-6.1-terra run achieves 62.4% (paired t-test p-value of 0.006). MACU also outperforms on the average rubric score, raising it from 82.0 to 92.3 (paired t-test p-value < 0.0001).

Replan budget (W3, Q2)

One unit corresponds to one applied replan decision (i.e., a DAG update), and we’ve corrected the "each edit" sentence accordingly. In Tab. 2, "B=1" meant the initial DAG with no replans, so we relabel the rows "single agent", "initial DAG only", "B=5" and "B=10".

OSWorld version (W4, Q3)

OSWorld-Verified is indeed the version we use, with the 361-task split (excluding Google Drive tasks). The versioning does not explain the gap. No existing implementation can reproduce the reported Qwen3.5 or Qwen3.6 results (e.g., see https://github.com/xlang-ai/OSWorld/issues/441 and https://github.com/xlang-ai/OSWorld/pull/448), so to the best of our knowledge, ours represents the highest scoring implementation available.

Ablation scope (W5)

All ablations use the official OSWorld test_small split as our development split. A full-scale replan ablation is unfortunately beyond our compute budget, but to do a test of whether the replanning ablation generalizes, we can examine the Odysseys task trajectories based on replans used, which gives a lower bound on the success rates of this ablation on the full Odysseys set:

Max replans usedOpus managerQwen3.6-27B manager
012.5%18.0%
122.5%22.0%
532.0%30.5%
734.0%32.0%

For other experiments, we rely mostly on this smaller ablation set for cost reasons (and also as it is the official OSWorld small subset) but note that results are generally intuitive and work well at scale. We agree that procuring and validating design decisions on a larger development set can further improve the performance of MACU on the full benchmarks, which we leave for future work.

60.4% (W6, Q4)

This was a typo and should be 60.0% which we’ve corrected.

Odysseys statistics (W7, Q5)

We have now included these numbers in Tab. 7. Opus manager MACU vs. single-agent has a p-value of 5×10−125 \times 10^{-12} and Qwen3.6-27B manager MACU vs single-agent has a p-value of 6×10−116 \times 10^{-11}.

References and layout (W8, W9)

We manually verified all references again and have updated this in the latest version. The blank space is a result of the whitespace preferences of the LaTeX / ICLR template and we decided not to modify it.

The cited systems achieving scores of 72.7% and 77.45% were evaluated on OSWorld, not Odysseys. Furthermore, the author's new evaluation does not cover the full Odysseys benchmark, nor does it include comparisons against strong baselines on Odysseys. I believe that each benchmark should be compared with strong baselines to ensure a fair and meaningful assessment. Given these limitations, I maintain my original score.

Thank you for reading our rebuttal! gpt-6.1-terra is a strong baseline: it's close to SOTA on OSWorld 2.1, and lies on the performance-cost pareto frontier.

We evaluate it on Odysseys-hard as it achieves close to 100% on the easy/medium tasks, and also close to 100% on OSWorld, so we do not think there is value in measuring on those benchmarks (as MACU and single-agent would achieve close to 100% and not be distinguishable). Given the cost of running these experiments, we opted to show results on Odysseys-hard as it has hard and long-horizon tasks that show the clearest delta for the benefits of MACU. We would appreciate the reviewer's consideration of these details.

If a single-agent system already achieves nearly 100% success on OSWorld, the motivation for MACU in terms of improving success rate on this benchmark becomes questionable. MACU may still offer benefits in efficiency or cost, but these would need to be demonstrated experimentally. Moreover, the claimed near-perfect performance is not supported by concrete evaluation results. I do not think this is a sufficient justification for omitting comparisons with strong baselines.

Reviewer_R2vgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MACU, a multi-agent framework where a manager breaks computer-use tasks into a DAG, runs subtasks in parallel, and updates the plan as results come in. MACU improves performance across several benchmarks, especially on long-horizon tasks. Ablations examine the impact of replanning, model strength, parallelism, and retries.

Critical strengths

Simple and effective multiagent design. The manager-worker setup is simple, and updating the DAG as workers collect new information is interesting. The large gains on Odysseys suggest that this design is useful for longer computer use tasks.

Good ablations. The paper studies the replanning budget, worker and manager models, parallelism, and pass@k. These experiments help explain where the gains come from. The experiments using Qwen3.6-27B as both manager and worker are also useful. They show that the improvement is not entirely due to adding a stronger manager.

Critical weaknesses

Limited evidence with stronger CUA workers. All main experiments use Qwen3.6-27B as the worker. Table 3 shows gains across all five tested Qwen3.5 and Qwen3.6 workers, which is useful evidence. GPT-5.4 and Claude Opus 4.6 are tested as managers, but the paper does not evaluate them as CUA workers. The paper should at least test Claude Opus 4.8 or later and GPT-5.5 or later as workers to see whether the gains hold with stronger models. Each single-agent baseline should use the same worker model, tools, and budgets. Appendix A reports results below the Qwen3.5 release numbers. The paper should clarify whether differences in the agent implementation or evaluation settings account for this gap. OSWorld success reaches 48.5%, and the improvement is not statistically significant in the paper's paired analysis. The score is far below leading reported results, e.g. a 90.2% from Intelligence-Indeed Agent, although some details need to be aligned.

Missing comparisons with other multiagent CUAs. The single-agent and retry baselines are useful, but the paper does not compare MACU experimentally with other multiagent CUAs. CoAct-1 is one example of relevant prior work. Such comparisons would help show what the adaptive DAG adds over existing coordination methods. Differences in worker models, tools, coding actions, and inference budgets should be accounted for.

Questions for authors

Which benchmark tasks were used to refine prompts, choose worker models, or set the replanning budget? Were any of these tasks, including the 36 OSWorld test_small tasks, included in the final evaluation? If so, can the authors report scores with those tasks excluded?

Can the authors provide results using Claude Opus 4.8 or later and GPT-5.5 or later as CUA workers, with single-agent baselines using the same worker model, tools, and budget?

How does MACU compare with other multiagent CUAs under comparable models, tools, and budgets?

Results on OSWorld 2.1 would provide useful additional evidence on longer desktop tasks.

1 comment

We thank the reviewer for their detailed review. We are glad that they appreciated MACU’s simple and effective manager-worker design, and recognized that the large gains on Odysseys highlight the effectiveness of our design for long-horizon computer use tasks. We are also glad that the reviewer found our ablations meaningful. We address specific questions below with new experimental results, and will incorporate all feedback in the next version of the paper.

Stronger workers

Unfortunately, running GPT-5.4 or Opus 4.6 as subagents is prohibitively expensive for us (we estimate ~\13pertaskaveragedacrossbenchmarks,i.e., 13 per task averaged across benchmarks, i.e., ~\\11K for this experiment). To address the reviewer’s concern however, we run MACU with gpt-6.1-terra with xhigh reasoning (a cheaper but close-to-frontier model) as both the subagent and the manager on the 109 hard Odysseys tasks, as these are the ones that latest models struggle most on (we find that gpt-6.1-terra achieves close to perfect on easy/medium difficulty tasks). We find that MACU's gains carry over to this stronger model: MACU achieves a perfect success rate of 74.3%, and a single-agent gpt-6.1-terra run achieves 62.4% (paired t-test p-value of 0.006). MACU also outperforms on the average rubric score, raising it from 82.0 to 92.3 (paired t-test p-value < 0.0001).

OSWorld score

MACU's 48.5% reflects the multi-agent setup with a Qwen3.6-27B agent backbone, which is significantly weaker than many frontier models. MACU also targets a different cost regime from running a frontier model serially: MACU costs about \0.25perOSWorldtask,comparedto0.25 per OSWorld task, compared to \\3.17 if we used Opus 4.6 as a single agent. On Odysseys it reaches 34.0% at \0.98vs44.50.98 vs 44.5% at \\2.5 for Opus 4.6.

Gap to the Qwen release

We use OSWorld-Verified (361 tasks) with a 1.5-hour limit, and timed-out single-agent runs score zero (25% of runs, Tab. 8). Unfortunately no existing implementation can reproduce the reported Qwen3.6-27B results on OSWorld (e.g., see https://github.com/xlang-ai/OSWorld/issues/441 and https://github.com/xlang-ai/OSWorld/pull/448), and to the best of our knowledge, our implementation represents the highest scoring one available.

Prior multi-agent CUAs

See the overall response. The controlled comparison on our axis is the same-backbone single agent, and the specialized systems differ in backbone, tools and action space.

Tuning tasks

We developed the prompts on the 36-task OSWorld split that we ran ablations on. The prompt includes several generic examples created by the authors (including the notes on Google Sheets / Hertz) which are common consumer tasks and not necessarily benchmark specific. Prompts and ablations iterated over the 36-task OSWorld test_small split. We now additionally include results on OSWorld excluding those tasks in the appendix. After removing these tasks, MACU maintains a lead over single agent on OSWorld (46.6% vs. 43.2% success rate).

We do not have benchmark specific changes, and we use the same setup and prompts for OSWorld, Online-Mind2Web, Odysseys, and WebTailBench. These examples do not drive the gains of MACU. MACU actually solves 0 of the 27 paired Odysseys tasks mentioning CryptPad or Hertz, and excluding these raises the success rate from 34.0% to 39.5%. Other benchmarks do not include these mentions.

Long horizon desktop use

We tested OSWorld 2 with Qwen3.6-27B as a single agent model. However, even in this setting, it scores a 0% success rate across all tasks, making it difficult to get a non-trivial baseline result using open weight models. We were not able to test OSWorld 2 (108 tasks at a 500-step budget) with stronger API-based models due to limited budget, but agree that it would be a promising direction for future work (especially with stronger open weight releases, which can be directly slotted into the MACU framework).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall response

We thank all reviewers for their detailed feedback. We are glad that the reviewers found the design simple and effective (R2vg, TZEk), that MACU is a valuable exploration of multi-agent systems in computer use (DFgb, BjaT). We appreciate that the reviews highlighted the strong results of MACU on long-horizon tasks and the comprehensiveness of our paper and experiments (DFgb).

We ran the suggested experiments which we believe address the remaining reviewer concerns. Our results demonstrate that MACU maintains its advantage even when we swap in stronger, more recent frontier models (which is simple given the model-agnostic setup that we propose). We have updated the PDF to incorporate these results and other feedback.

Handling timeouts (BjaT, TZEk)

We experiment with forcing an answer for Odysseys runs that do not produce a final answer due to timeout, and re-judge these runs. We use the same final-summary prompt from the original Odysseys benchmark for this. Under this setting, MACU maintains an edge over the single agent baselines. The single-agent setup achieves a perfect success rate of 23.5%, while MACU achieves 34.0%, which is statistically significant with a two-sided paired t-test p-value of 0.0037.

In addition, the Odysseys experiment we ran with gpt-6.1-terra model (detailed below) has no timeouts within the 3 hour allowance, highlighting that MACU’s gains are not due to the single agent timing out more frequently.

Stronger workers and baselines (R2vg, DFgb)

Reviewers requested a comparison with a stronger, more recent CUA model. We run gpt-6.1-terra (a cheaper near-frontier model) with xhigh reasoning as both the manager and subagents for MACU, and compare against a single-agent gpt-6.1-terra run. We run both on all 109 hard Odysseys tasks, with the same step, replan, and time budgets for this experiment. MACU retains its gains with this stronger model as a backbone, achieving a significantly better perfect success rate of 74.3%, versus 62.4% for the single agent (two-sided paired t-test p = 0.006), and also raises the average rubric score from 82.0 to 92.3 (p < 0.0001). No run in either setup reached the 3-hour limit, so timeout handling does not affect this comparison.

Replan budget (DFgb, BjaT, TZEk)

We rephrased the replan budget definition to be clearer (one replan, i.e. a DAG change costs one unit of budget). We have updated the paper throughout and any stale references.

Comparison to prior multi-agent CUAs (R2vg, TZEk)

An additional concern raised was regarding the novelty of MACU. We note that "multi-agent" is a somewhat overloaded term with several meanings, including (1) role specialization and (2) task decomposition and inference-time scaling. Our work focuses on the latter. Agent S2, CoAct-1 and InfantAgent-Next focus on the former and are heavily specialized, which makes an apples-to-apples comparison difficult:

  • Agent S2 routes actions through a "Mixture-of-Grounding" of specialist experts alongside separate manager and worker modules.
  • CoAct-1 comprises three heterogeneous agents (Orchestrator, Programmer, GUI Operator) with a programmatic action space.
  • InfantAgent-Next similarly integrates specialized tool, code and vision modules.
  • LLMCompiler and Anthropic's multi-agent blogpost are focused on non-CUA domains (they use tool calls and text rather than GUI), and Agent S2 is serial. We have updated Sec. 2 with these details.

Running them on our backbone would require significant re-engineering, while running them as released would confound backbone, grounding modules and action space with orchestration. All three are also serial pipelines, so they do not isolate and study decomposition and parallelization like we do.

A core motivation of MACU is to avoid such specialization. A single manager plus homogeneous subagents makes the single-agent baseline with the same backbone the controlled comparison, and any CUA model can be dropped in unchanged. Sec. 2 in our updated paper expands on this.

We thank all reviewers again for their valuable comments and suggestions. We believe that our responses and proposed changes address the concerns brought up in each review, but please let us know if there is anything else we can further clarify before the end of the discussion phase!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.