acceptodds
Withdrawn from ICLR 2027

Multi-Agent LLM Serving Is a Full-Stack Problem: A Small Step Towards Algorithm–System Co-Design

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models are increasingly deployed as multi-agent systems (MAS) that solve a task by coordinating many LLM invocations through structured control flow such as branching, debate, and iterative refinement. This capability imposes a serving cost that current infrastructure is ill-equipped to absorb: algorithm researchers iterate on MAS designs rapidly and optimize for task accuracy, whereas serving systems evolve slowly and cannot be specialized to each workflow, so MAS run on general-purpose engines blind to workflow structure. We introduce a unified graph representation that casts existing and emerging MAS algorithms as typed graphs, turning opaque multi-agent executions into analyzable system workloads. On this representation we develop two families of optimization: static patterns, recurring motifs that a runtime accelerates without changing the algorithm, and dynamic strategies, small structural changes that trade accuracy for large efficiency gains. On real GPUs shared-prefix reuse cuts prefill latency by –, confidence-gated early exit reaches – speedup at – fewer tokens, and sparse expert dispatch lowers cost roughly for points of pass@1. Crucially, the prefill win does not survive end to end: decode is – of latency here, so end-to-end speedup stays at parity (–, every CI covering ), making phase-awareness a prerequisite for co-design. We intend this work as a guide for designing hardware-friendly MAS algorithms and a step toward algorithm–system co-design for LLM serving.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
multi-agent system

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_2CfrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates the serving efficiency of multi-agent LLM systems and highlights the gap between algorithm design and serving-system optimization. The authors propose a unified typed workflow graph representation, together with a phase-aware cost model that distinguishes prefill and decode costs. Based on this framework, the paper categorizes optimization techniques into two groups: static execution-level optimizations that preserve workflow semantics, and dynamic graph-level strategies that trade computational efficiency against task quality. The authors summarize six static patterns and five dynamic strategies, and evaluate several of them on mathematical reasoning and code generation benchmarks. The experiments show that prefix caching can improve time-to-first-token latency without necessarily providing meaningful end-to-end speedups, while strategies such as early exit and model downscaling can reduce computational costs with varying accuracy trade-offs. The paper also observes that some multi-agent workflows do not outperform a simple single-model baseline.

Critical strengths

S1. A useful perspective on algorithm–system co-design.

The paper provides a structured perspective on multi-agent serving by representing different workflows as typed graphs and analyzing their costs across prefill, decode, and execution dependencies. This framework offers a useful way to organize existing optimization techniques and understand their applicability to different workflow structures.

S2. Informative empirical observations, including negative results.

The paper evaluates both local optimization benefits and end-to-end performance, revealing that improvements in prefill efficiency may not translate into meaningful overall speedups. It also shows that some multi-agent configurations fail to outperform simpler single-model baselines. These observations provide useful practical insights and highlight the importance of evaluating actual serving costs rather than focusing exclusively on task accuracy or isolated performance metrics.

Critical weaknesses

W1. Limited technical novelty and lack of a substantial new optimization method.

The paper introduces a unified workflow graph and a phase-aware cost model, but most of the discussed optimization techniques, including prefix caching, early exit, heterogeneous model assignment, and dynamic routing, have already been explored in prior work. Representing workflows as computation graphs and applying Amdahl's Law to analyze performance improvements are also well-established ideas. More importantly, the experiments primarily apply existing techniques or simple hand-designed heuristics rather than introducing a new optimization algorithm or system. The paper does not demonstrate how its unified framework can automatically select, combine, or jointly optimize different strategies. The authors also acknowledge in Section 6 that the optimizations are manually designed, leaving automatic co-design as future work. Therefore, while the paper offers useful empirical observations, the independent technical contribution of the proposed framework remains unclear.

W2. Limited experimental validation of the broader full-stack co-design claims.

Although the paper discusses six static optimization patterns and five dynamic strategies, only a subset is directly evaluated, while the remaining techniques are mainly supported by existing literature and analytical arguments. Furthermore, the experiments focus on a single model family, mathematical reasoning and code generation benchmarks, and single-request serving. They do not evaluate multi-tenant concurrency, real tool interactions, or more complex production-level agent workflows. The limited end-to-end benefits of prefix caching are also closely related to the decode-dominated workloads used in the experiments. It remains unclear how broadly these findings generalize to workloads with different prefill/decode characteristics. Overall, the current experimental evidence is insufficient to establish the practical effectiveness and generality of the proposed framework for full-stack algorithm–system co-design.

Questions for authors

Q1. Novelty and independent contribution of the proposed framework.

Could the authors provide a concrete example where the unified graph representation and cost model enable a new optimization decision or capability that cannot be directly obtained from existing techniques? How does the proposed framework go beyond organizing and explaining established optimization methods?

Q2. Generalizability of the experimental findings.

How well do the reported findings extend to more realistic multi-agent serving scenarios involving concurrent requests, tool interactions, or more complex workflow dependencies? In particular, would the conclusions about prefix caching change substantially for workloads with a higher proportion of prefill computation?

Q3. Optimization strategy selection and quality trade-offs.

The experiments show that techniques such as early exit and model downscaling can reduce computational costs but may also degrade accuracy on harder tasks. How should the proposed framework guide the selection and configuration of these strategies under different workloads and accuracy requirements? Is there evidence that such decisions can be made more effectively using the proposed cost model than with existing heuristics?

Reviewer_atJbOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that multi-agent LLM serving needs algorithm–system co-design, and offers a vocabulary for it: MAS algorithms are cast as typed workflow graphs G = (N, E, Op) with a phase-aware cost model (prefill + decode per node) and an Amdahl bound on end-to-end speedup.

Critical strengths
  • Paired McNemar tests, bootstrap CIs, an independent replicate run, full per-cell tables, and candid statements that few-point changes are within noise.
  • The finding that a direct Qwen3.5-9B call (0.775) beats a five-role pipeline (0.658) and matches a dense five-expert ensemble (Table 2) is exactly the baseline the community under-reports
  • The Amdahl framing is a useful corrective.
Critical weaknesses
  • The technical contribution is largely a taxonomy of existing work. The typed-DAG view of agent programs is already the premise of Parrot, Ayo, Agentix, GPTSwarm, and LangGraph; each static pattern is an existing system (SGLang, Parrot, Pie, InferCept, KVCOMM) and each dynamic strategy an existing method (FrugalGPT, DAAO, MaAS, OneFlow, AgentArk).
  • Co-design is claimed but never measured. The paper’s definition of co-design (§4.2, “Interactions”) is that the two families interact — downscaling breaks prefix sharing, pruning removes the calls that made caching worthwhile. No experiment applies a static pattern and a dynamic strategy jointly.
  • Half the catalogue is unvalidated. Patterns III, V, VI and Strategies IV, V “rest on the graph analysis and prior systems” (Limitations). Three of six patterns and two of five strategies are described but never run
Questions for authors

What is single-call Qwen3.5-9B accuracy on each math benchmark, and how does a fixed T=1 debate compare to early exit? What does prefix reuse deliver end-to-end in the non-thinking, 512-token regime used for the dynamic studies, where prefill share should be substantial? Can you report throughput or goodput under concurrency for at least the fan-out pattern? Is there any experiment combining a static pattern with a dynamic strategy, so the “interactions” claim is measured?

Reviewer_u5SZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Many LLM applications now run as multi-agent workflows. The paper's main idea is that the serving system should be able to see the shape of these workflows. It describes a workflow as a graph. The nodes are model calls, tool calls, and routers. The edges carry data between nodes and record whether the data moves from CPU to GPU or back. Two operators handle control flow. One merges parallel branches, and one decides which branch runs next. The paper adds a simple cost model on top. Each model call has two phases: prefill, which reads the prompt, and decode, which writes the output one token at a time.

The paper, using this graph, lists six "static patterns" and five "dynamic strategies". A static pattern is a shape in the graph that an existing serving trick can exploit without changing the answer. A dynamic strategy can change the graph and thus the answer, e.g., by stopping early when agents agree or by moving some nodes to a smaller model. Multi-Agent Debate and MetaGPT are worked through as examples.

Critical strengths

The main point is simple, and the main negative result is well measured. The paper says that what matters is not just the shape of the workflow but how much time each phase takes. That is right. The paper also checks whether a single model call does as well as the whole multi-agent setup on code, and finds that it does.

Critical weaknesses
  1. The paper mostly sorts existing work into categories and then confirms what those categories predict. The graph (model and tool calls as nodes, with control operators) is what existing systems/works (e.g., Ayo, and AFlow) already work with. The edge tags and the router node are somewhat new in the description, but it seems they are never used in any experiments. All six patterns and all five strategies are credited to earlier systems or methods. The experiments test existing mechanisms, but none of these is new, and the paper builds no runtime or optimizer (Section 6 only describes one).

  2. Some claims in the paper may not hold if the set up is changed, for instance: (1) The code study compares against a weak stand-in for MetaGPT. The authors' own five-prompt pipeline and five experts that differ only in their prompts, all on a 9B model. The pipeline loses to a single call even though its QA role sees every test case. That shows that this pipeline is weak. I think it does not support the conclusion that "some multi-agent structure buys nothing over a single call." (2) The paper reports that prefix caching makes the first token arrive up to 2.42x sooner but leaves total latency unchanged. But I think two setup choices can lead to this result. First, every call runs in thinking mode and writes 2k to 12k tokens, so decode takes 95% to 99% of the time. With a single request, prefix caching speeds up only prefill, so even a perfect cache could cut total time by a few percent at most. Second, the server handles one request at a time. On a shared server, prefill work saved on one request frees the GPU for other requests, so the server can handle more requests per second. That throughput gain is the main reason serving systems cache prefixes, and the paper never measures it.

Questions for authors
  1. Does prefix reuse improve throughput when many requests run at once (say 16 to 64 debates in parallel) in the same setup?
  2. Did the debate agents run at the same time in the dynamic studies? If not, how do the early-exit and downscaling speedups change when they do run in parallel on the same GPU?
  3. What does the cost model predict that plain Amdahl's law does not? Did you check any prediction about KV memory held on the GPU or stalls during tool calls against a measurement?
  4. How did you pick the 45-word threshold and the keyword list for the gate and the router, and on what data? If you picked them on MBPP itself, those results are tuned on the test set.
  5. Are the edge tags or the router node used anywhere in the analysis or the experiments?
Reviewer_GE2MOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that serving systems for multi-agent LLM workflows should use the structure of the workflow. It considers a workflow as a typed graph of model calls, tool calls, and control operators. Its cost model splits the time of each model call into prefill and decode. With Amdahl's law, this model bounds the end-to-end gain of an optimization. The paper lists static patterns and dynamic strategies. Experiments with Qwen3.5 models measure prefix reuse, early exit, and node downscaling on math benchmarks. The paper also evaluates code workflows on MBPP.

Critical strengths

S1. The authors report the experimental results carefully, with confidence intervals and a repeated run.

S2. The code study shows that no multi-agent workflow is more accurate than a single model call.

Critical weaknesses

W1. The technical contribution is limited.

  • The static patterns and dynamic strategies are not novel. Each is drawn directly from existing systems and prior methods.
  • GPTSwarm, Parrot, and Ayo already represent LLM workflows as graphs.

W2. Experiments are not comprehensive.

  • The core claim of algorithm–system co-design is not supported by experiments. The paper defines co-design as "planning both under one cost model", but no experiment plans the two families together.
  • The paper expects prefix reuse to be worthwhile with long shared contexts and short answers, but it does not measure this setting.

W3. The paper does not discuss related refereed work comprehensively.

  • Murakkab (OSDI '26) already uses the structure of agentic workflows to choose the model.
  • ReConcile (ACL '24) and S$^2$-MAD (NAACL '25) terminate a multi-agent discussion early when a consensus is reached.
  • Adaptive-Consistency (EMNLP '23) stops sampling when the sampled answers agree strongly enough.

W4. The presentation needs improvement.

  • The text in Figure 1 is too small to read, especially in the diagrams and the legend.
  • Section 5 mostly lists numbers that are already in the tables, rather than explaining the main takeaways.
Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.