- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
AgentServeSim presents a simulation framework for evaluating program-aware LLM agent serving policies. Built on LLMServingSim 2.0, it introduces program-level execution tracking and persistent KV cache management to jointly simulate KV retention, routing, and scheduling across multiple LLM calls and tool executions. Experiments show that AgentServeSim predicts real-serving program completion time with a small mean absolute percentage error and correctly identifies the best policy across three evaluated settings.
Critical strengths
- Agent serving simulation is a very timely and important topic.
Critical weaknesses
- Limited abstraction that cannot support a broad range of optimizations. The paper essentially integrates a specific set of existing agent serving techniques into a simulator. However, agent serving is still a rapidly evolving research area, and the proposed abstraction seems too restrictive to accommodate emerging optimizations. For example, the abstraction only provides interfaces for KV cache management upon turn arrival and completion, while many existing optimizations involve offloading, prefetching, and migration during tool execution. Consequently, researchers exploring new optimization techniques would still need to make substantial modifications to the simulator, significantly limiting the practical value of the proposed abstraction in reducing development effort.
- Insufficient justification for the proposed approach compared to serving-engine-based simulators. Simulators built directly on existing serving engines, such as SGLang Simulator and Tair-KV HiSim, seem to offer a more flexible and practical approach for evaluating emerging agent serving optimizations. These simulators can reuse the scheduling logic of existing serving engines while replacing actual GPU computation with simulated execution. As a result, researchers who have already implemented their optimizations in a serving engine can evaluate them in a simulator with minimal additional engineering effort. In contrast, the proposed simulator requires researchers to reimplement their optimizations within its restrictive abstraction, potentially introducing substantial engineering overhead. Moreover, the paper does not discuss or compare against this category of simulators, leaving the advantages of its design choices unclear.
Questions for authors
No
4 comments
Response to Reviewer 1oUh(2/2)
W2. Insufficient justification for the proposed approach compared to serving-engine-based simulators.
Engine-based simulation is useful when an optimization already exists in the target engine and can reuse its execution code. AgentServeSim provides a complementary route: a common program-policy interface whose implementations run in simulation and real vLLM, including coordination across engine instances. For the evaluated policies, implementations require 86–469 lines, and in our automated porting study all six Continuum and InferCept ports were generated within 2.5 minutes and ran without edits.
Design choice. Agent-serving optimizations typically change the very scheduling and KV cache management logic that an engine-based simulator reuses. Continuum, for example, changes how the engine orders calls and when it evicts a program's KV cache. An engine-based simulator can therefore simulate an optimization only after it is implemented in that engine. Existing optimizations modify different engines and versions, and comparing them in one engine-based simulator requires porting them to that engine. AgentServeSim instead models the engine, as Vidur (Agrawal et al., 2024), LLMServingSim 2.0 (Cho et al., 2026), APEX (Lin et al., 2024) and Frontier (Feng et al., 2026) do. It models vLLM. Supporting another engine, such as SGLang, would require modeling its timing and execution semantics. Policies using operations supported by that backend could then be compared across engine models in simulation.
Porting effort (Appendix H). To measure the porting effort, we gave an LLM, Claude Sonnet 5.5, the released vLLM code of Continuum and InferCept, along with our interface and documentation, and requested three independent ports of each. The Continuum ports have 94–112 lines and took about 40 seconds and $0.15 each to generate; their simulated mean JCT is within 0.8% of real vLLM. The InferCept ports have 204–282 lines and took under 2.5 minutes and \$0.26 each; their simulated mean JCT is within 1.1–1.8% of real vLLM.
Comparison with the SGLang Simulator and Tair-KV HiSim (§2, Appendix K). These simulators reuse SGLang's scheduler and cache code. The SGLang Simulator supports trace replay, synthetic workloads, and hierarchical KV caching, with a logical-time mode and a real-time (blocking) server mode. Its trace replay releases requests at recorded times. In blocking mode, an external client can preserve call and tool dependencies using real-time execution; preserving them in logical-time simulation requires additional workload coordination. Their documented interfaces do not expose program-level policy control over retaining, offloading, or prefetching a paused program's KV cache during tool execution. Both simulate a single worker, so routing, work stealing, and KV cache migration between instances cannot be evaluated, and neither reports validation on agent workloads.
To isolate replay's effect, we kept our engine model unchanged and fixed call arrivals to those recorded in the real Stock run (B200, Llama-3.1-70B, 50 SWE-Bench programs at 0.10 programs/s). Across six policies, mean JCT error rose from 2.9% to 18.0%, and Kendall τ against real vLLM fell from 1.00 to −0.07. Replay also predicted Evolved, the real winner, to be slower than Stock. We have added the comparison with engine-based simulators to §2, the porting study to Appendix H, and the replay experiment to Appendix K (Table 20).
Thank you for the detailed rebuttal and additional experiments. The clarification of the policy interface and the new experiments address several of my original concerns, particularly regarding support for dynamic KV management and implementation effort.
However, I remain concerned about the generality of the proposed abstraction beyond its predefined operations and its advantages over simulators that directly reuse serving engines. I acknowledge the improvements and will raise my score from 1 to 2.
Official Comment by Authors
Dear Reviewer 1oUh,
Thank you for reading our response and for raising your score. We briefly address the two remaining points.
Generality beyond the predefined operations. The operations are engine-level primitives: routing, ordering, admitting, and preempting calls, and protecting, releasing, discarding, offloading, restoring, prefetching, or migrating KV caches. Policies decide when and how to apply them, so new decision rules over these operations need no simulator changes; every optimization we evaluated is expressed this way. An optimization that needs additional engine behavior, such as a new transfer path, requires extending the engine model; that capability then becomes available to every policy through the interface. §3.4 states this boundary.
Advantages over engine-based simulators. The two approaches are complementary. Engine-based simulation directly reuses optimizations implemented in its target engine. AgentServeSim provides a common engine model for comparing and combining independently developed policies from different engines and engine versions. It also supports multi-instance routing and KV cache migration between instances, and preserves how each policy changes subsequent call arrivals and KV availability across tool waits.
We are happy to clarify anything further before the discussion period ends.
Best regards, The Authors
Response to Reviewer 1oUh(1/2)
Thank you for raising the abstraction and engineering-effort concerns. The implemented interface supports offloading, prefetching, and migration during tool execution. Policies evaluated in the paper already use these capabilities, and two new policies (102 and 118 lines) use them without simulator changes, with mean-JCT predictions within 5% of real vLLM. We clarify this interface and explain where a common policy model is useful relative to engine-based simulation.
W1. Limited abstraction. KV cache management seems limited to turn arrival and completion, so offloading, prefetching, or migration during tool execution would need simulator changes.
The interface is not limited to turn arrival and completion. In AgentServeSim, an agent-serving optimization is written as a policy, and policies make serving decisions at every engine step, including during tool execution. A policy can therefore offload a paused program's KV cache, prefetch it before the tool is predicted to return, or migrate it to another instance. Policies in the paper already do all three, and two new policies validate them on real vLLM.
Our abstraction separates policy decisions from engine enforcement. Policies observe program and instance state, such as tool timing, KV cache location, memory availability and load. They make serving decisions, such as routing, scheduling, admission, preemption, and KV cache management. Policies express their decisions through callbacks, including one that runs at every engine step, even while a program waits for a tool (Appendix H, Table 16). Our simulator models the serving engine and carries out these decisions. It accounts for prefix sharing, protection, admission constraints, and transfer completion.
All the optimizations we evaluate, existing and new, use this interface without simulator changes. The optimizations evaluated in the paper, each written as a policy (§5.1, Appendix I), already offload, prefetch, and migrate KV caches during tool waits: Autellix PLAS uses accumulated program service, and its full MLFQ variant swaps preempted calls' KV caches to host memory; InferCept uses KV size and estimated tool wait to choose retention, offloading, or discarding; Continuum uses tool history for TTL retention; and full SAGA uses load and KV cache location for routing, prefetching, and stealing. Some policies proposed by our LLM-driven search (§5.2) use rules absent from these baselines; Evolved, for example, delays admitting a call until its uncached prompt fits in memory. We also implemented two new policies, Prefetch and Forced stealing (102 and 118 lines), that offload, prefetch, and migrate KV caches during tool execution, and ran both on real vLLM (Llama-3.1-70B, B200 GPUs, 50 SWE-Bench programs):
We clarified the interface and its extension boundary in §3 (especially §3.4) and Appendix H, which also reports the two policies and their real-vLLM validation (Tables 16 and 18).