- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MACU, a multi-agent setup for computer use. A manager model splits a task into a graph of subtasks. Up to four computer use agents then work on ready subtasks at the same time, each in its own virtual machine. When a subtask finishes, the manager can change the graph: add, cancel or rewrite subtasks, pass findings to later subtasks, and pass files between machines. At the end, the manager writes the final answer and picks one machine as the final state. All workers use Qwen3.6-27B. The manager is Opus 4.6 or Qwen3.6-27B. MACU improves success rates on OSWorld (43.8 -> 48.5), Online-Mind2Web (52.2 -> 55.6), WebTailBench-v2 (20.8 -> 29.5) and Odysseys (8.5 -> 34.0)
Critical strengths
- The method is simple and does not depend on a specific worker model. Better workers can be plugged in directly.
- The gain on the long-horizon Odysseys benchmark is large. Most of it stays when the same Qwen model is used as the manager (32.0 vs. 34.0).
- The paper reports costs, latency and negative results honestly, for example the slower runs on Online-Mind2Web. Tables 8-10 give detailed statistics, and all prompts are included.
Critical weaknesses
- The single-agent baseline is limited by the time cap, so the comparison is not fair. On Odysseys, 55% of single-agent runs stop at the 3-hour limit, and 25% do so on OSWorld. Only 10 of 200 Odysseys runs reach the 200-step limit, and the average run has only about 98 actions. So the "plateau" in Fig. 3 comes from cutting runs off, not from the agent being stuck. In Fig. 6 (bottom), the single-agent curve is still going up at 180 minutes. MACU also runs four machines at once and uses 365 VM-minutes per Odysseys task, versus 145 for the single agent (Table 8). In addition, when MACU hits the time limit, the manager still writes a final answer from the partial results. This adds about 7 points at the end of the Fig. 6 curve. The paper does not say whether the single agent also gets a final answer when it times out. Please add a single-agent run with a longer time limit, or with the same total VM time. Please also give the single agent a final answer step at timeout.
- Some baselines are needed to show where the gain comes from: the strong manager or the multi-agent graph. On OSWorld, 42 of the 61 tasks solved only by MACU use just one worker (Table 10). In Table 13, the Opus instructions spell out almost every key press. Three baselines are missing. (a) Opus writes a detailed plan, and one Qwen agent carries it out step by step. Fig. 5 (N = 1) is close to this, but it uses only 45 tasks. (b) Several parallel single-agent runs with a judge that picks the best one, at the same compute, such as Behavior Best-of-N in Agent S3. This work is not cited. The pass@8 comparison uses only 36 tasks and a 4B worker. (c) Existing multi-agent computer use systems, such as Agent S2 or CoAct-1. Also, when the same model is both manager and worker, the gain on short tasks is near zero: +1.1 on OSWorld, and +0.2 on Online-Mind2Web in Table 1 but -1.4 in Table 7.
- The ablations use a small setting that may not be representative. All ablations on the planning budget, manager and worker run on 36 OSWorld tasks, mostly with a 4B worker. On this subset, the 27B worker gains 19.5 points (47.2 -> 66.7, Table 3). On the full OSWorld, it gains only 4.7. The main claims are about long web tasks, but there is no budget or follow-up ablation on Odysseys or WebTailBench. The definition of B is also unclear (see Minor 1). So the claim that ongoing replanning drives the gains is not well supported. Table 10 describes the runs but does not show cause, as the authors also note.
- The novelty is limited. A manager that plans a task graph and sends work to parallel workers is a known design, for example LLMCompiler, Anthropic's multi-agent research system, and Agent S2. The paper says it treats partial observability as a first-class problem. But file passing between machines is used in only 0-2.5% of tasks (Table 9). Most of the gain seems to come from retries, variants and passing findings to later subtasks. Please make clearer what is new for computer use compared with these systems.
Questions for authors
- When the single agent hits the time limit, does it produce a final answer? How does it do with a longer time limit?
- In Fig. 2, the manager missed cheaper fares ($1,169 and $1,284), but the judge still passed the "cheapest fare" rubric. How often does the Odysseys judge make this kind of mistake? Did you check some judgments by hand?
- How were the manager prompts developed? Were they tuned on tasks from the test benchmarks? Some prompt parts look benchmark-specific (for example tab curation, CryptPad and Hertz examples).
We thank the reviewer for their detailed review. We are glad that the reviewer found MACU simple and independent of the underlying backbone model, and recognized its gains on long-horizon benchmarks. We address specific questions below with new experimental results, and incorporate all feedback in the updated paper.
Time limit (W1, Q1)
We experiment with forcing a final answer wherever a run lacks one, using the final-summary prompt from Jang et al. (2026). Under this setting, MACU maintains an edge over single agent, achieving a perfect success rate of 23.5% for single-agent vs. 34.0% for MACU (statistically significant with a paired t-test p-value of 0.0037).
While we agree that the time limit benefits multi-agent systems, this is a fundamental limitation of single-agent setups, and is part of our core argument that CUAs should move towards being trained and deployed as multi-agent systems. We also believe that a 3 hour time limit is generous for these tasks, which the gpt-6.1-terra experiment (see overall response) never hits.
Missing baselines (W2)
We agree the gain on short tasks is limited, which fits our claim that gains concentrate on decomposable tasks, and we removed "across all benchmarks" from Sec. 4.2.
Ablation setting (W3)
We’ve updated the paper to reflect that these tasks are our dev split, which is the official OSWorld "test_small" set. We rely mostly on this smaller set for cost but note that results are generally intuitive and work well at scale. We now include results on OSWorld excluding those tasks in the appendix. After removing these tasks, MACU maintains a lead over single-agent (43.2% vs. 46.6% success rate).
Unfortunately, a full-scale budget ablation on the full OSWorld set is beyond our budget, but we conducted an analysis on the existing MACU runs to identify Odysseys success rates by the number of replans used. This gives some lower bounds: at most 0 replans achieves 12.5%, at most 5 gives 32.0%, and at most 7 gives 34.0%, which highlights that replanning is indeed helpful.
Novelty (W4)
LLMCompiler and Anthropic's multi-agent blog are focused on non-CUA domains (tool calls and text rather than GUI), and Agent S2 is serial. We updated Sec. 2 with these details. Regarding the CUA-specific capabilities, such as file passing, we highlight that this is actually the least-used of three channels (App. A.1). Other novel features we introduced in this work are also prevalently useful in achieving strong performance: VM continuation is used in 25-35% of tasks (Tab. 9), VM forks on 18-49% (Tab. 9), and forwarded findings on 11-59% (Tab. 9).
Judge (Q2)
For this specific example, the rubric checks that the "final response explicitly names which site offered the cheapest fare and states the price", which the judge applies as written (it is not able to extract the global minimum price from live sites). We use the same judge from Odysseys, which reported a high human agreement of κ=0.849. Repeated judging agrees on ~95% of rubrics for both setups. We also manually inspected several judge results and did not identify any substantial false positives or negatives.
Prompts (Q3)
We developed the prompts on the 36-task OSWorld split that we ran ablations on. The prompt includes several generic examples created by the authors (including the notes on Google Sheets / Hertz) which are common consumer tasks and not necessarily benchmark specific.
We do not have benchmark specific prompts, and use the same setup for OSWorld, Online-Mind2Web, Odysseys, and WebTailBench. The examples do not drive the gains of MACU. MACU solves 0 of the 27 paired Odysseys tasks mentioning CryptPad or Hertz, and excluding them actually raises the success rate from 34.0% to 39.5%.