- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Maia is a dynamic serving system for Agent-as-a-Service workloads in which multiple agent types share a common LLM backbone but differ in adapters, prompts, tools, and runtime state. Maia supports runtime reconfiguration of agent instances using two mechanisms: consistency-aware reuse of prompt KV caches with selective recomputation of context-sensitive tokens, and hybrid runtime-state migration that overlaps KV transmission with recomputation. It further formulates instance assignment as an online optimization problem that balances unmet service demand against reconfiguration cost. Testbed experiments and trace-driven simulation show reduced reconfiguration latency and lower aggregate utility loss than static and myopic dynamic allocation baselines.
Critical strengths
The paper addresses a concrete systems bottleneck specific to stateful agent serving. Prompt initialization and runtime-state migration are meaningful costs that are not captured by ordinary stateless LLM serving, and the proposed reconfiguration mechanisms directly address these costs.
Critical weaknesses
-
The end-to-end baselines are too weak to establish the claimed systems advantage. The main comparison is against static allocation and a myopic dynamic policy that pays the full reconfiguration cost. However, recent systems already address closely related parts of this problem with substantially stronger mechanisms. KVFlow introduces workflow-aware KV-cache eviction and prefetching for multi-agent workflows based on future agent execution structure [1], while PBKV extends this direction to dynamic agent workflows by predicting future agent invocations and managing cache retention/prefetching accordingly [2]. On the scheduling side, Agent.xpu develops flow-aware scheduling and fine-grained preemption for concurrent agentic LLM workloads under heterogeneous resources [3]. These systems do not solve exactly the same problem as MAIA, in particular, MAIA uniquely emphasizes cross-agent-type reconfiguration and runtime-state migration, but they represent substantially stronger points of comparison than SAM and the myopic DAM baseline. A direct comparison, or at least a controlled adaptation of these mechanisms to MAIA's workload model, would be important to establish the incremental benefit of MAIA's proposed reconfiguration and allocation design.
-
Quality preservation of cross-agent KV reuse is evaluated too narrowly. The main accuracy-retention experiment uses a small MMLU sample rather than realistic stateful agent tasks. Since the method intentionally reuses KV representations whose preceding context differs between agent roles, semantic correctness is central to the approach and should be validated at the agent-task level.
[1] Pan, Zaifeng, et al. "KVFlow: Efficient prefix caching for accelerating LLM-based multi-agent workflows." Advances in Neural Information Processing Systems 38 (2026): 126246-126265.
[2] Zheng, Haoyu, et al. "Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management." arXiv preprint arXiv:2605.06472 (2026).
[3] Wei, Xinming, et al. "Agent. xpu: Efficient scheduling of agentic llm workloads on heterogeneous soc." arXiv preprint arXiv:2506.24045 (2025).
Questions for authors
-
Can the authors provide an end-to-end comparison with recent workflow-aware caching or dynamic agent-scheduling systems under matched workloads?
-
How does consistency-aware KV reuse affect task success on actual multi-turn/tool-using agent workloads rather than MMLU accuracy alone?
-
How closely does the trace-driven simulator reproduce the reconfiguration and scheduling behavior of a larger physical deployment under concurrent sessions?
3 comments
Thank you again for your insightful comments. Our rebuttal clarifies MAIA’s relationship to KVFlow, PBKV, and Agent.xpu, provides KVFlow integration results and MT-Bench scores, and discusses simulator validation against the physical deployment. We would welcome your feedback on these responses. Please let us know if further clarification or additional evidence would be helpful. We would be happy to supplement our response and revise accordingly.
Thank you again for your careful review and thoughtful comments. Our rebuttal clarifies MAIA’s relationship to existing systems and provides KVFlow integration results, MT-Bench scores, and simulator validation against the physical deployment. We would appreciate any further feedback when convenient. If additional clarification or revisions would be helpful, we would be happy to address them.
We clarify differences from related systems and provide additional results on AutoGen integration, conversational quality, and simulator fidelity.
We thank the reviewer for the thoughtful feedback and positive assessment.
Answer to Q1 and W1: Relationship to existing systems and integration benefits.
(1) Differences from existing work. KVFlow uses workflow structure to guide KV-cache eviction and prefetching, while PBKV predicts future agent invocations for cache management. Agent.xpu coordinates execution and preemption across heterogeneous SoC accelerators. MAIA addresses a complementary problem: allocating instances across agent types under changing demand and reducing the initialization and state-migration costs of reconfiguration. We will clarify these differences in decision scope and execution setting.
(2) Additional evaluation with KVFlow. To demonstrate MAIA's benefits within an existing agent-serving stack, we evaluated KVFlow with a SGLang backend on an 8×L20 testbed, using the same dynamic workload traces with and without MAIA.
Enabling MAIA increases throughput by 2.03×, reduces P95 latency by 65.2%, and lowers SLO violations by 43.07 percentage points. These results demonstrate additional benefits when MAIA is integrated with an existing orchestration framework and serving engine.
Answer to Q2 and W2: Multi-turn output quality.
Beyond the MMLU results in Figure 8, we additionally evaluate MAIA on MT-Bench to assess two-turn conversational quality. Using LLaMA3-8B and GPT-4o as the judge, we average two independent ratings for each turn's response:
The score differences are 0.125 and 0.112 points, respectively. MAIA's scores remain close to full recomputation across both turns, providing additional evidence that selective prompt KV reuse preserves conversational output quality in the evaluated setting.
Answer to Q3: Simulator fidelity and concurrent sessions.
We validate the simulator against an eight-GPU physical deployment in Appendix F. The testbed uses eight NVIDIA L20 GPUs, serving LLaMA3-8B agents through vLLM with one GPU per instance. We replay ten-hour WildChat request traces covering two agent domains.
Relative to the physical MAIA deployment, the simulator's throughput error is 5.4%, and its P50/P90/P99 latency errors are 7.8%/7.6%/8.9%. These results validate its aggregate performance predictions in the evaluated deployment. The physical experiment also demonstrates MAIA's practical benefit: throughput increases by approximately 2×, while P99 latency decreases by 64.2%.