- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors present a collaborative multi-agent framework "Muffin" based on the use of a shared buffer. From my understanding their method roughly works as follows:
- An "orchestrator" LLM reads the question and decides if it answers directly or if it needs a team. In the latter case, it will spawn 2-3 subagents and assign those subagents precise roles/subtasks
- Each agent then runs for a fixed number of steps. At each step, the agent reads the most recent notes left in the buffer and performs its reasoning. The model then produces i) a short summary for itself ii) what (if anything) it will write in the buffer denoted iii) private notes
- Once all reasoning is finished, a summarizer agent takes as input the question + every agent's final summary and picks the best answer
The method is asynchronous in the number of steps. An agent doesn't wait for another to finish, it just reads from the buffer when its next step starts. If a note lands mid-step, it is only seen at the next step.
The authors empirically validate their method on multiple benchmarks (MMLU, GPQA, OlymMath, Chess) and show that their method outperforms majority voting and multi-agent debate baselines across 17/20 of their selected benchmarks.
All in all I think the paper is interesting and the method is promising, but I find that their headline numbers overclaim a bit. I would raise my score if
- The authors include (at least) one novel baseline based on a different cooperative protocol
- The authors either revise/remove the token count reduction claim or make it in a fairer scenario (see weakness 2 for more details)
Critical strengths
-
The method is simple, interesting and seems to work rather well. I believe the approach to be promising work.
-
The authors provide many ablations allowing the reader to better understand the method and which parts of it are most crucial to its success.
Critical weaknesses
Critical weakness 1: the authors do not evaluate their method against any other cooperative multi-agent systems.
- This makes it hard to determine if the collaborative approach of their method is the driving factor in their accuracy results. It would be interesting to see how their method stacks up against e.g. GroupThink (https://arxiv.org/abs/2505.11107), Mixture-of-agents (https://arxiv.org/abs/2406.04692)
- Relatedly, https://arxiv.org/abs/2510.13903 studies the agent-count vs. communication-bandwidth trade-off and should be discussed in related work, since it directly relates to the results in Fig 4 and Fig 6.
Critical weakness 2: the claim on token reduction feels cherry picked
- By design, Muffin forces summarization and reads from a small sized buffer which reduces total token count. The multi-agent debate protocol proposed by Du et al. has every agent see the full previous responses of all other agents at each round which by design would inflate token count. This feels like an unfair comparison. Thus, I find the headline "Muffin improves accuracy by 19.2 accuracy points while using 77.4% fewer tokens, without requiring full-trace sharing" to be quite misleading.
- Moreover, in Figure 3, it can be seen that both single agent and majority vote baselines require far less tokens than Muffin for the same "step" count.
- It could be interesting to either: i) report token efficiency only via generation/CoT (see related question) which would make the comparison more fair ii) compare token efficiency to one of the baselines mentioned above.
Questions for authors
- What roles/strategies/subtasks the agents typically end up using for the problem decomposition? I am curious how this changes across task domains also.
- Would it be possible to show a version of Table 1 which includes token counts?
- Would be interesting to see how performance changes when a larger model is used as orchestrator but the subagents are still rather small
- How do token efficiency results hold up if one looks only at output (e.g. CoT/generation) tokens, not input + output?
- What is the default value of used across experiments?
5 comments
Question 3: Use a large model for the orchestrator and summarizer, and for subagents use a small model
This is an interesting suggestion, and we will run an experiment on OlymMATH-E. In this setting, only the orchestrator and the summarizer use the larger gemma-4-31B-it, with all subagents using the smaller gemma-4-E4B-it. We compare this with results from the Gemma-4 model family, where all components use the same model. This setting raises accuracy from 59.3% to 63.0%, while total token cost stays nearly the same as using only the small model. The smaller model uses about 3x more output tokens per task, mostly on private thinking, while its shared summaries are shorter, which is why the summarizer’s input is smaller. The gap with only using a 31B model is still large, showing that the sub-agents’ reasoning capacity is the main bottleneck.
| Orchestrator / Summarizer | Subagents | Accuracy | Orchestrator tokens (in/out) | Summarizer tokens (in/out) | Subagent tokens (in/out) | Total tokens |
|---|---|---|---|---|---|---|
| gemma-4-E4B-it | gemma-4-E4B-it | 0.593 ± 0.012 | 0.9k / 1.9k | 1.8k / 0.7k | 14.5k / 64.7k | 84.5k |
| gemma-4-31B-it | gemma-4-E4B-it | 0.630 | 0.9k / 0.8k | 1.8k / 0.5k | 14.3k / 67.2k | 85.5k |
| gemma-4-31B-it | gemma-4-31B-it | 0.850 ± 0.010 | 0.9k / 0.8k | 4.3k / 0.5k | 15.2k / 20.1k | 41.9k |
Question 4: How do token efficiency results hold up if we only look at the completion token
Here we report input and output tokens separately for both Multi-Agent Debate and Muffin. The 77.4% token reduction stated in the paper comes from a single 10-step run on OlymMATH-E with gemma-4-E4B-it, so it may not generalize. We instead use the average over all runs reported in Table 1, which includes 4 models x 5 benchmarks, and got a similar result of 74.3% fewer total tokens. The results are shown in the table below. On output tokens alone, Muffin generates 1.6x more than Debate. This is expected from the orchestrator, the structure step’s format, and the summarizer. On the other hand, Muffin uses 16.7x fewer input tokens because we have a compact prompt format. If we consider this as cost, the commercial API usually prices output tokens at about 4-5x input tokens. Under this weighting, Muffin is still around 30% cheaper than Multi-agent Debate overall.
| Debate | MUFFIN | MUFFIN vs. Multi-Agent Debate | |
|---|---|---|---|
| Input | 403.9k | 24.1k | 94.0% fewer (16.7×) |
| Output | 59.5k | 95.0k | 59.6% more (1.60×) |
| Total | 463.4k | 119.1k | 74.3% fewer (3.9×) |
| Cost (output = 4× input) | 641.9k | 404.0k | 37.1% cheaper |
| Cost (output = 5× input) | 701.4k | 498.9k | 28.9% cheaper |
Question 5: What is the default of k?
We use by default in all experiments, so each agent is able to read the 10 most recent findings of other agents from the memory buffer. The only exception is the ablation on the number of shared findings we report in Section 4.3, where we set .
Weakness 1: No comparison against other cooperative multi-agent systems.
We thank the reviewer for this suggestion. We are adding Learning to Share (LTS)[1] as a new baseline, suggested by another reviewer, since it is the cooperative protocol closest to Muffin. We will discuss more to distinguish our work from Group Think, Mixture-of-Agents and LTS. We will add the reference you provided to the related work [2].
vs. Groupthink
Group Think runs one LLM as several thinkers that see each other’s partial generations token by token. Agents therefore share their full traces, and the method needs a modified decoding loop. Muffin agents share only the findings they choose to publish, and Muffin runs on any standard serving engine.
vs. Mixture-of-agents
MOA is layer-synchronous, each agent needs to wait for and read the complete responses of all agents in the previous layer. Muffin agents run asynchronously and exchange compact findings during reasoning rather than full responses after it.
vs. LTS
LTS and Muffin both share intermediate information, but they differ in what is parallelized and who controls the sharing. LTS runs K independent teams (same prompt, same agent) in parallel. Within each team, the execution remains sequential. After each worker's step, a learned controller decides whether to store the raw output, and only each team’s orchestrator can read the memory, using LLM call to select keys. Muffin instead parallelizes within a single team of agents with roles and subtasks defined by the orchestrator. And sharing is controlled by the agent itself, which is part of the generation, with no extra training or selection calls.
This table summarizes the key design differences between Muffin and the related approaches.
| Method | Shared Memory | Async | Query-adaptive team design | Training-free | Standard serving engine |
|---|---|---|---|---|---|
| Mixture-of-Agents | Previous-layer responses | ✖ | ✖ | ✓ | ✓ |
| Group Think | Token-level traces | ✓ | ✖ | ✓ | ✖ |
| LTS | Cross-team | ✓ | ✖ | ✖ | ✓ |
| Muffin | Within-team | ✓ | ✓ | ✓ | ✓ |
LTS as a Baseline
We evaluate LTS in our problem setting using the authors’ released code and their pretrained memory controller, and use the same backbone model as Muffin. Since our benchmarks are closed-book, we removed the WebSurfer agent from each team, leaving all other components unchanged. We first provide the comparison on the OlymMATH-E benchmark with gemma-4-31B-it. We also compare against two variants of LTS proposed in their paper. LTS-LLM replaces the trained controller with an LLM-as-a-judge that decides what to store, and LTS-AddAll stores everything in the shared memory without any filtering. As shown in the table below, Muffin achieves higher accuracy than all LTS variants while using fewer tokens in total. We also see that the trained memory controller version is a bit worse than the LLM-as-a-Judge heuristic. This might be because the controller was trained on AssistantBench, a tool-intensive web task, and may not transfer to reasoning tasks without retraining.
| Method | Avg. Accuracy | Avg. Input Token | Avg. Output Token |
|---|---|---|---|
| LTS | 82.5 ± 0.021 | 115.9K | 19.3K |
| LTS-LLM | 84.0 ± 0.014 | 108.7K | 18.7K |
| LTS-AddAll | 82.0 ± 0.021 | 107.6K | 18.1K |
| Muffin | 85.0 ± 0.010 | 20.4K | 21.5K |
References
[1] Fioresi, J., Kulkarni, P. P., Vayani, A., Wang, S., and Shah, M. Learning to Share: Selective Memory for Efficient Parallel Agentic Systems. Proceedings of the 43rd International Conference on Machine Learning. https://proceedings.mlr.press/v306/fioresi26a.html
[2] Michael Rizvi-Martel, Satwik Bhattamishra, Neil Rathi, Guillaume Rabusseau, Michael Hahn. Benefits and Limitations of Communication in Multi-Agent Reasoning. https://arxiv.org/abs/2510.13903
Question 1: Role choice from the orchestrator
We analyze the orchestrator-generated roles from Muffin runs in Table 1 on five benchmarks and four backbone models, with three agents and five reasoning steps per agent. In this experiment setting each team consists of three agents, one of which is always a Critic as required by the orchestrator's prompt, and the other two are chosen depending on the domain. The table below shows the share of each functional role among the agents assigned by the orchestrator. And here are the descriptions and examples of how we cluster the agents’ roles:
Question 2: Table 1 with token counts
This table reports the mean number of input and output tokens per task. We also take the average over the five benchmarks and compare the total token usage with Muffin.