- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes DeLM, a multi-agent framework in which parallel agents coordinate without a main orchestrating agent. Each agent has its own session and container. They share an append-only context and a task queue that enforces exclusive claims and shows each item's status.
The authors argue that existing multi-agent systems waste agent time in "bubbles": redundant work, waiting at synchronization barriers, and waiting on a main agent to relay results. They claim DeLM removes these bubbles.
Critical strengths
The problem of idle and redundant agent time in parallel agentic coding is practically relevant.
The paper reports latency and cost alongside accuracy. The execution timeline illustrations are informative.
Critical weaknesses
Main ones:
First of all, the major problem, I don't think the "decentralized" claim holds up. In line 91 the authors put blackboard-style systems under centralized orchestration, and in Appendix A the only difference they draw is that those systems have a control agent. But DeLM itself has one central task queue and one central shared context, which is basically a blackboard. Taking the controller out and having prompted agents do its job doesn't make the coordination decentralized. In practice it isn't leaderless either: the queue starts with "predefined, general preparation subtasks" (Fig. 4, Alg 1) with no explanation of where they come from, the prompt says "the first worker creates the initial queue" (line 1033), and when the queue runs out, "the agent that most recently completed a subtask" decides whether to add more (lines 180). That is a rotating central planner over central state, and I don't see how it really differs from the main-agent decomposition in Claude Code style orchestration.
Almost every experiment is DeLM against centralized systems, but there are basically no comparisons with other non-centralized approaches. The paper's own taxonomy (lines 80, Table 1) lists independent agents and multi-agent debate, yet neither shows up in the results. At the very least there should be a compute-matched baseline of n parallel ReAct agents with no communication and a clear rule for picking the final answer. Without that, the claim that DeLM is "both faster and more accurate … improving over the strongest baseline" (line 154) doesn't mean much, because the most obvious baseline is missing. The paper also brings up the straggler problem in debate-style systems (line 86) but never measures it statistically.
The comparison with the centralized baselines does not seem to be fair on resources. Each DeLM agent has "its own persistent session and workspace" (line 185) and "a separate container" (line 1031), while Claude Code/Codex main agents and their subagents all share one workspace. So DeLM gets n isolated environments, which removes file conflicts and resource contention and lets builds and tests run in parallel. The extra environment resources are never reported, and the cost numbers only count API spend. The authors should show this trade-off or give the baselines the same isolated setup.
DeLM seems to be "built on top of Codex and on top of Claude Code", so it isn't at the same abstraction level as Codex and Claude Code. Comparing it with a single copy of the harness mixes up the coordination mechanism with just running more copies, which contradicts the claim that "the only difference between them is how agents coordinate" (line 256). The fair comparison should be against other ways of coordinating multiple instances of the same harness.
The main results are questionable. Terminal-Bench 4.0 has 66 tasks, and the public leaderboards put GPT-6-Astra at about 58% and Claude Opus 5.5 at about 64%. On the authors' 10-task subset, the single-agent baselines already get 71.67% (Codex) and 81.67% (Claude Code). So the picked tasks look much easier than average even though they were chosen as "long-horizon." The tasks were also picked based on one baseline's latency (lines 234), DeepSWE is cut to 10 tasks too, and the std across three runs is 10-13 points. So cherry-picking can't be ruled out. The authors should run at least a much bigger pre-defined set and add significance tests. Also, Claude Code (subagent) dropping from 71.67% to 50.00% on DeepSWE (Table 3) is a huge drop and looks like a misconfigured baseline.
It looks like the acc gains mostly come from the prompt, not from coordination. In the Fig. 5 ablation, removing any single mechanism (even the whole shared context) bring speed back to about 1×, but acc barely moves. So coordination explains the speedup but not the roughly 10-point accuracy gain, which goes against the conclusion that "these gains come from coordination" (lines 479). The DeLM prompt is very long and full of engineering advice that has nothing to do with coordination (how to handle performance-constrained work, validation steps, "Derive necessary checks from the original task", measurement rules), while the baselines use default prompts. A single-agent baseline with the DeLM prompt minus the coordination parts is needed. Some of the prompt items, like "first-call path" and "timing measurements", look tailored to the performance and measurement tasks for Terminal Bench / Deep Swe. The paper doesn't say what tasks the prompt was tuned on and has no held-out set, so overfitting to the 10 test tasks is a real possibility. On top of that, the n=4 prompt is different from the n=2 one (lines 1204), which is also strange.
Also, the metric "Cost/Submission" makes the cost look lower than it actually is. The user only needs one solution in the end, and the n submissions all come from the same shared context, so they are highly correlated and not really n independent solutions. Dividing total cost by n (lines 262) artificially shrinks the cost. Cost/Task should be the number that matters, and by that measure DeLM usually costs more than the single-agent baseline. and when the text says DeLM is cheaper, it only compares against the subagent baselines.
And some other minor stuff:
Claude Code already has Agent Teams (https://code.claude.com/docs/en/agent-teams) a long time ago, where teammates share a task list with pending/in-progress/completed states and dependencies, claim unassigned tasks themselves (with file locking to avoid races), and message each other directly. That is very close to DeLM's idea: queue with exclusive claims, peer status, and shared findings, and it's available in the same Claude Code version the authors build on. The paper doesn't mention or compare with it, and Table 1 marks Claude Code as lacking peer-status awareness, which isn't accurate.
The name "Decentralized Language Models" is misleading. This is a multi-agent coordination framework built on top of existing agent harnesses, not a language model. Nothing is trained or changed at the model level.
Questions for authors
-
It looks like the shared context can only grow by appending (Sec. 2.1, 4.4), so how does it stay under 15K tokens (Table 5)? Is that from the character limits in the prompt, from truncation, or just because the selected tasks are short? How does it grow with longer tasks and more agents, and what happens when it gets past what the model can handle well? Line 203 also says later reads "emphasize new publications". Does each agent actually read the whole context, and is the read cost in Table 5 calculated that way?
-
How does DeLM perform if you keep increasing the number of agents to 8, 16, and beyond? At what point do queue contention, duplicated work, and the growing shared context start to hurt?
-
Every agent submits its own solution, so how do you pick the one final answer for a task? If there is a selection step, how accurate is it?
-
How exactly is accuracy computed? If each run scores 10 pass/fail tasks with one submission, per-run accuracy should move in 10% steps, and a three-run mean should be a multiple of 3.33%. Several reported numbers don't fit that. For example, Codex's 71.67 +/- 11.55 on TerminalBench only works with 5% steps (e.g., runs of 65/65/85), and some entries, such as Codex (subagent) at 71.67 +/- 11.69 and DeLM (n=2, Opus 5.5) at 93.33 +/- 3.33, can't be reproduced with any simple step size. Is there partial credit, and how is it assigned? For DeLM, is accuracy the average over the n submissions, or does a task count as solved if any submission passes? If it's the latter, the comparison is basically pass@n vs. pass@1? Please report the per-run scores for every method.
-
For SWE-bench Verified, which subset did you use (all 500 or a sample)? Why switch to AOrchestra's harness here instead of Claude Code or Codex? And average latency there is only about 60–130 seconds, which doesn't fit the paper's focus on "long-horizon" tasks. What is this experiment meant to show?
13 comments
Author Response to Reviewer cqYi (2/10)
(Continued from the previous comment) W1: The 'decentralized' claim
No agent waits on a coordinator. From the board's event log, we measured how long agents are blocked in read --wait, and split each team's agent-time into working, blocked, and finished (already submitted while a teammate is still working). Idle is the share of wall-clock time in which no agent is working:
Results are mean ± standard deviation over three repeats.
The first two columns are per agent; the next three split the team's total agent-time and sum to 100%. Results are mean ± standard deviation over three repeats.
| Method | Blocked (% of own session) | Blocked (min / agent) | Working (% of team time) | Blocked (% of team time) | Finished (% of team time) | Idle (% of wall-clock) |
|---|---|---|---|---|---|---|
| TerminalBench-4.0, GPT-6-Astra | ||||||
| DeLM () | 1.57 ± 0.62 | 0.25 ± 0.12 | 95.33 ± 3.43 | 1.55 ± 0.62 | 3.12 ± 2.97 | 0.00 ± 0.00 |
| DeLM () | 3.21 ± 0.77 | 0.45 ± 0.16 | 93.19 ± 0.52 | 3.13 ± 0.75 | 3.68 ± 1.10 | 0.00 ± 0.00 |
| TerminalBench-4.0, Opus 5.5 | ||||||
| DeLM () | 4.42 ± 2.16 | 2.61 ± 1.57 | 84.30 ± 2.82 | 4.01 ± 1.87 | 11.69 ± 0.94 | 0.33 ± 0.47 |
| DeLM () | 3.58 ± 2.34 | 2.00 ± 1.66 | 78.17 ± 0.19 | 3.04 ± 2.18 | 18.79 ± 2.07 | 0.00 ± 0.00 |
| DeepSWE v1.1, GPT-6-Astra | ||||||
| DeLM () | 1.43 ± 0.37 | 0.23 ± 0.09 | 96.84 ± 0.96 | 1.42 ± 0.36 | 1.74 ± 0.65 | 0.00 ± 0.00 |
| DeLM () | 2.78 ± 0.30 | 0.36 ± 0.02 | 93.33 ± 0.96 | 2.67 ± 0.27 | 4.00 ± 0.73 | 0.00 ± 0.00 |
| DeepSWE v1.1, Opus 5.5 | ||||||
| DeLM () | 3.93 ± 0.85 | 1.18 ± 0.30 | 87.28 ± 1.52 | 3.59 ± 0.68 | 9.13 ± 0.85 | 0.03 ± 0.06 |
| DeLM () | 3.20 ± 0.90 | 1.00 ± 0.29 | 86.09 ± 1.74 | 2.89 ± 0.80 | 11.02 ± 1.09 | 0.00 ± 0.00 |
DeLM agents spend 1.4–4.4% of their session blocked, and there is no working agent for at most 0.33% of the wall-clock time. For comparison, on the task in Fig. 3, the main agent of Claude Code's native subagents spends 73% of its time waiting on its subagents.
To further test how DeLM differs from main-agent orchestration, we built a centralized variant that keeps DeLM's worker prompt, shared context and expand/apply tools, and adds a controller that assigns the work, integrates the results, and returns one final submission. This is a shared board with a control agent. It uses the same number of agents as DeLM (): one controller and three workers. On TerminalBench-4.0 (Results are mean ± standard deviation over three repeats):
| Method | Avg. Acc (%) | Avg. Latency (min) | Cost/Task | Cost/Submission |
|---|---|---|---|---|
| GPT-6-Astra | ||||
| Codex | 71.67 ± 11.55 | 26.91 ± 0.91 (1.00×) | \0.71 | \0.71 |
| Centralized () | 80.00 ± 10.00 | 19.31 ± 1.62 (1.39×) | \1.89 | \1.89 |
| DeLM () | 81.67 ± 12.33 | 13.16 ± 1.43 (2.05×) | \2.27 | \0.57 |
| Claude Opus 5.5 | ||||
| Claude Code | 81.67 ± 2.89 | 130.75 ± 6.27 (1.00×) | \0.23 | \0.23 |
| Centralized () | 86.67 ± 5.77 | 89.05 ± 7.73 (1.47×) | \1.63 | \1.63 |
| DeLM () | 89.17 ± 5.20 | 52.51 ± 4.43 (2.49×) | \0.91 | \0.23 |
Accuracy differs by only 1.67 and 2.50 points, within the variation across repeats, but the centralized variant takes 46.7% and 69.6% longer at a similar Cost/Task. With everything else fixed, the controller alone accounts for the gap between a 1.39–1.47× and a 2.05–2.49× speedup. The centralized baselines we compare against therefore leave latency on the table by making agents wait on a controller.
Author Response to Reviewer cqYi (3/10)
W2: Comparisons with non-centralized approaches
We compare DeLM with independent agents and with Claude Code Agent Teams, and we measure the straggler cost of debate-style synchronous rounds.
Compute-matched independent agents. For every task and repeat, our baselines already launch two runs concurrently on the same server, and the two runs never communicate. Together they form a system of two independent agents, the same number of agents as DeLM (), and it spends more per task than DeLM in all four settings. We compare under three rules for returning one answer: avg@1 returns a random submission (the Avg. Acc of our main tables), Fastest returns the first submission and needs no selector, and pass@2 counts a task as solved if any submission passes, an oracle upper bound for any selection rule.
Latency under avg@1 is the mean agent latency; latency under Fastest is the time until the first agent finishes. Cost/Task sums the costs of all agents. Results are mean ± standard deviation over three repeats.
| Method | avg@1 Acc (%) | avg@1 Latency (min) | Fastest Acc (%) | Fastest Latency (min) | pass@ (%) | Cost/Task |
|---|---|---|---|---|---|---|
| TerminalBench-4.0, GPT-6-Astra | ||||||
| Codex, 2 independent agents | 71.67 ± 11.55 | 26.91 ± 0.91 | 76.67 ± 20.82 | 23.03 ± 0.60 | 86.67 ± 11.55 | \1.41 |
| DeLM () | 85.00 ± 13.23 | 17.47 ± 1.82 | 86.67 ± 15.28 | 16.83 ± 2.31 | 86.67 ± 15.28 | \1.49 |
| TerminalBench-4.0, Claude Opus 5.5 | ||||||
| Claude Code, 2 independent agents | 81.67 ± 2.89 | 130.75 ± 6.27 | 80.00 ± 10.00 | 111.65 ± 3.14 | 90.00 ± 0.00 | \0.46 |
| DeLM () | 93.33 ± 5.77 | 64.77 ± 5.03 | 93.33 ± 5.77 | 47.86 ± 3.68 | 93.33 ± 5.77 | \1.17 |
| DeepSWE v1.1, GPT-6-Astra | ||||||
| Codex, 2 independent agents | 88.33 ± 2.89 | 16.57 ± 0.39 | 90.00 ± 0.00 | 14.38 ± 0.38 | 90.00 ± 0.00 | \0.36 |
| DeLM () | 98.33 ± 2.89 | 13.29 ± 0.98 | 96.67 ± 5.77 | 13.05 ± 0.89 | 100.00 ± 0.00 | \1.65 |
| DeepSWE v1.1, Claude Opus 5.5 | ||||||
| Claude Code, 2 independent agents | 71.67 ± 2.89 | 49.58 ± 3.11 | 80.00 ± 10.00 | 43.53 ± 3.16 | 86.67 ± 5.77 | \1.20 |
| DeLM () | 88.33 ± 2.89 | 32.98 ± 2.47 | 90.00 ± 0.00 | 29.22 ± 1.98 | 90.00 ± 0.00 | \0.53 |
Under both avg@1 and Fastest, DeLM () is more accurate than two independent agents in all four settings, returns its answer sooner, and costs less per task. Its avg@1 accuracy even exceeds the oracle pass@2 of independent agents in three of the four settings (93.33% vs. 90.00%, 98.33% vs. 90.00%, and 88.33% vs. 86.67%) and is within 1.67 points in the fourth (85.00% vs. 86.67%). The gain is therefore not explained by having more submissions: in three settings, even a perfect selector over independent agents would not reach DeLM's accuracy without any selection.
Straggler cost of synchronous rounds. We measured the straggler cost you mention on the same independent runs. In a synchronous round, an agent that finishes early waits for the slowest one. For agents finishing at times , a barrier leaves a fraction of agent time idle, and the round takes times as long as the average agent:
| Setting | n=2 Idle (%) | n=2 Debate speedup (upper bound) | n=2 DeLM speedup | n=4 Idle (%) | n=4 Debate speedup (upper bound) | n=4 DeLM speedup |
|---|---|---|---|---|---|---|
| TerminalBench-4.0, GPT-6-Astra | 11.44 | 0.88× | 1.54× | 18.92 | 0.81× | 2.05× |
| TerminalBench-4.0, Claude Opus 5.5 | 12.33 | 0.87× | 2.02× | 23.66 | 0.75× | 2.49× |
| DeepSWE v1.1, GPT-6-Astra | 11.14 | 0.88× | 1.25× | 19.11 | 0.80× | 1.34× |
| DeepSWE v1.1, Claude Opus 5.5 | 11.03 | 0.88× | 1.50× | 19.39 | 0.79× | 1.57× |
Idle is the share of agent time spent waiting for the slowest agent in one synchronous round. The debate speedup bound is the average agent's latency divided by the round time, for one round alone; speedups are relative to a single agent.
A single synchronous round already leaves 11–12% of agent time idle with two agents and 19–24% with four. With Claude Opus 5.5 on TerminalBench-4.0, each of four agents waits 41.14 minutes on average. Debate pays this cost in every round.
Debate also targets a different goal. It was proposed to improve answer quality (Du et al., 2023; Liang et al., 2023), not to reduce latency: every agent produces a complete solution in every round, so a debate run can never finish sooner than its slowest agent's single attempt. The table above therefore bounds the speedup of debate over a single agent: at most 0.87–0.88× with two agents and 0.75–0.81× with four, for one round alone. DeLM (n=2) instead reaches 1.25–2.02×, because it has no rounds and therefore no barrier.
(Continued in the next comment.)
Author Response to Reviewer cqYi (1/10)
Thank you for your thorough review. We appreciate your recognition of the practical relevance of idle and redundant agent time in parallel agentic coding, and the value of reporting latency, cost, and accuracy alongside execution timelines.
We address your concerns below: (W1) whether DeLM is truly decentralized; (W2–W4) the missing non-centralized baselines, resource fairness, and the level at which DeLM is compared; (W5) task selection; (W6) the role of the prompt; and (W7) the cost metric, followed by your minor points and questions. To do so, we added seven new experiments and analyses: a centralized controller with the same workers (W1), compute-matched independent agents and a straggler analysis for debate (W2), Claude Code Agent Teams (W2), 42 additional tasks chosen by a fixed rule from public data (W5), baselines with a matched prompt (W6), and n=8 and n=16 agents (Q2).
Several of the new experiments (Agent Teams in W2, the 62 tasks in W5) run the baselines with a matched prompt: the DeLM prompt with its coordination instructions removed and all of its task-solving guidance kept verbatim. W6 describes how we built it.
We hope this addresses your concerns and would be grateful if you could consider updating your score in light of these improvements.
W1: The 'decentralized' claim
By decentralized, we mean decentralized control: no agent assigns the others' work or integrates their results, so no agent waits on a coordinator. We also respectfully argue that no system is purely decentralized, and that what matters is the form of coordination. DeLM's agents coordinate asynchronously. DeLM centralizes what agents store, not who decides: the board stores entries and the queue makes atomic claims, but neither decides which agent works on what or what enters a final answer.
Each agent reads the board directly, works from its own view of it, and decides for itself what to claim and when to submit. The board's only job is to keep these views in sync, and we show that using it as the synchronization primitive substantially lowers latency without hurting accuracy (see the centralized variant below). DeLM is also more accurate than the single-agent harness, and this holds when we control for the prompt (W6).
Our paper does not intend to discredit existing blackboard systems for using a centralized blackboard, but to emphasize an important point: “how” a blackboard is used for coordination and synchronization matters a lot more than what the community previously thought. This finding is one of our main novelties.
Existing systems synchronize more than they need to, which wastes agent time, and our ablations show how much each mechanism matters.
We call coordination centralized when one agent assigns the others' work and integrates their results, so that the others wait on it. DeLM has no such agent.
The three points you raise. None of them gives one agent authority over the others or makes the others wait:
(Continued in the next comment.)