acceptodds
Under review as a conference paper at ICLR 2027

AgentServeSim: Simulation-Driven Policy Design for Agent Serving

Abstract

Serving large language model (LLM) agents requires coordinating cache man- agement, routing, and scheduling across model calls separated by tool execution. Simulating these policies is difficult because their decisions change subsequent call arrivals, retained context can be shared across programs, and cache reuse depends on both placement and resource availability. AgentServeSim addresses these cou- pled requirements through a program-level execution model that preserves history and cache state across LLM calls and enforces policy actions within common resource constraints. It derives arrivals from simulated completions, accounts for shared prefixes without duplicating their memory cost, and checks cache availabil- ity at admission. Across 20 comparisons with real serving, AgentServeSim predicts mean program job completion time (JCT) with 4.74% mean absolute percentage er- ror. It also identifies the same winning policy as real serving among six candidates in all three evaluated settings. Using simulation to evaluate LLM-generated KV- management and scheduling code, we discover a policy that achieves 1.64–2.04× speedup in real serving over a default baseline using first-come-first-served (FCFS) scheduling and least-recently-used (LRU) KV cache eviction, and 1.25–1.45× over the strongest evaluated alternative in each setting.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM serving, LLM agents, systems simulation, LLM-driven policy optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1oUhOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AgentServeSim presents a simulation framework for evaluating program-aware LLM agent serving policies. Built on LLMServingSim 2.0, it introduces program-level execution tracking and persistent KV cache management to jointly simulate KV retention, routing, and scheduling across multiple LLM calls and tool executions. Experiments show that AgentServeSim predicts real-serving program completion time with a small mean absolute percentage error and correctly identifies the best policy across three evaluated settings.

Critical strengths
  1. Agent serving simulation is a very timely and important topic.
Critical weaknesses
  1. Limited abstraction that cannot support a broad range of optimizations. The paper essentially integrates a specific set of existing agent serving techniques into a simulator. However, agent serving is still a rapidly evolving research area, and the proposed abstraction seems too restrictive to accommodate emerging optimizations. For example, the abstraction only provides interfaces for KV cache management upon turn arrival and completion, while many existing optimizations involve offloading, prefetching, and migration during tool execution. Consequently, researchers exploring new optimization techniques would still need to make substantial modifications to the simulator, significantly limiting the practical value of the proposed abstraction in reducing development effort.
  2. Insufficient justification for the proposed approach compared to serving-engine-based simulators. Simulators built directly on existing serving engines, such as SGLang Simulator and Tair-KV HiSim, seem to offer a more flexible and practical approach for evaluating emerging agent serving optimizations. These simulators can reuse the scheduling logic of existing serving engines while replacing actual GPU computation with simulated execution. As a result, researchers who have already implemented their optimizations in a serving engine can evaluate them in a simulator with minimal additional engineering effort. In contrast, the proposed simulator requires researchers to reimplement their optimizations within its restrictive abstraction, potentially introducing substantial engineering overhead. Moreover, the paper does not discuss or compare against this category of simulators, leaving the advantages of its design choices unclear.
Questions for authors

No

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1oUh(1/2)

Thank you for raising the abstraction and engineering-effort concerns. The implemented interface supports offloading, prefetching, and migration during tool execution. Policies evaluated in the paper already use these capabilities, and two new policies (102 and 118 lines) use them without simulator changes, with mean-JCT predictions within 5% of real vLLM. We clarify this interface and explain where a common policy model is useful relative to engine-based simulation.

W1. Limited abstraction. KV cache management seems limited to turn arrival and completion, so offloading, prefetching, or migration during tool execution would need simulator changes.

The interface is not limited to turn arrival and completion. In AgentServeSim, an agent-serving optimization is written as a policy, and policies make serving decisions at every engine step, including during tool execution. A policy can therefore offload a paused program's KV cache, prefetch it before the tool is predicted to return, or migrate it to another instance. Policies in the paper already do all three, and two new policies validate them on real vLLM.

Our abstraction separates policy decisions from engine enforcement. Policies observe program and instance state, such as tool timing, KV cache location, memory availability and load. They make serving decisions, such as routing, scheduling, admission, preemption, and KV cache management. Policies express their decisions through callbacks, including one that runs at every engine step, even while a program waits for a tool (Appendix H, Table 16). Our simulator models the serving engine and carries out these decisions. It accounts for prefix sharing, protection, admission constraints, and transfer completion.

All the optimizations we evaluate, existing and new, use this interface without simulator changes. The optimizations evaluated in the paper, each written as a policy (§5.1, Appendix I), already offload, prefetch, and migrate KV caches during tool waits: Autellix PLAS uses accumulated program service, and its full MLFQ variant swaps preempted calls' KV caches to host memory; InferCept uses KV size and estimated tool wait to choose retention, offloading, or discarding; Continuum uses tool history for TTL retention; and full SAGA uses load and KV cache location for routing, prefetching, and stealing. Some policies proposed by our LLM-driven search (§5.2) use rules absent from these baselines; Evolved, for example, delays admitting a call until its uncached prompt fits in memory. We also implemented two new policies, Prefetch and Forced stealing (102 and 118 lines), that offload, prefetch, and migrate KV caches during tool execution, and ran both on real vLLM (Llama-3.1-70B, B200 GPUs, 50 SWE-Bench programs):

Policy and settingWhat the policy doesJCT error, mean / p95
Prefetch (1 GPU, 0.03 programs/s)Offloads a paused program's KV cache and restores it before predicted tool completion−0.4% / −2.5%
Forced stealing (2 GPUs, 0.12 programs/s)Routes all calls to one instance; the other takes waiting calls and their KV caches whenever its queue is empty+4.9% / +0.4%

We clarified the interface and its extension boundary in §3 (especially §3.4) and Appendix H, which also reports the two policies and their real-vLLM validation (Tables 16 and 18).

AuthorsOpenReview ↗

Response to Reviewer 1oUh(2/2)

W2. Insufficient justification for the proposed approach compared to serving-engine-based simulators.

Engine-based simulation is useful when an optimization already exists in the target engine and can reuse its execution code. AgentServeSim provides a complementary route: a common program-policy interface whose implementations run in simulation and real vLLM, including coordination across engine instances. For the evaluated policies, implementations require 86–469 lines, and in our automated porting study all six Continuum and InferCept ports were generated within 2.5 minutes and ran without edits.

Design choice. Agent-serving optimizations typically change the very scheduling and KV cache management logic that an engine-based simulator reuses. Continuum, for example, changes how the engine orders calls and when it evicts a program's KV cache. An engine-based simulator can therefore simulate an optimization only after it is implemented in that engine. Existing optimizations modify different engines and versions, and comparing them in one engine-based simulator requires porting them to that engine. AgentServeSim instead models the engine, as Vidur (Agrawal et al., 2024), LLMServingSim 2.0 (Cho et al., 2026), APEX (Lin et al., 2024) and Frontier (Feng et al., 2026) do. It models vLLM. Supporting another engine, such as SGLang, would require modeling its timing and execution semantics. Policies using operations supported by that backend could then be compared across engine models in simulation.

Porting effort (Appendix H). To measure the porting effort, we gave an LLM, Claude Sonnet 5.5, the released vLLM code of Continuum and InferCept, along with our interface and documentation, and requested three independent ports of each. The Continuum ports have 94–112 lines and took about 40 seconds and $0.15 each to generate; their simulated mean JCT is within 0.8% of real vLLM. The InferCept ports have 204–282 lines and took under 2.5 minutes and \$0.26 each; their simulated mean JCT is within 1.1–1.8% of real vLLM.

Comparison with the SGLang Simulator and Tair-KV HiSim (§2, Appendix K). These simulators reuse SGLang's scheduler and cache code. The SGLang Simulator supports trace replay, synthetic workloads, and hierarchical KV caching, with a logical-time mode and a real-time (blocking) server mode. Its trace replay releases requests at recorded times. In blocking mode, an external client can preserve call and tool dependencies using real-time execution; preserving them in logical-time simulation requires additional workload coordination. Their documented interfaces do not expose program-level policy control over retaining, offloading, or prefetching a paused program's KV cache during tool execution. Both simulate a single worker, so routing, work stealing, and KV cache migration between instances cannot be evaluated, and neither reports validation on agent workloads.

To isolate replay's effect, we kept our engine model unchanged and fixed call arrivals to those recorded in the real Stock run (B200, Llama-3.1-70B, 50 SWE-Bench programs at 0.10 programs/s). Across six policies, mean JCT error rose from 2.9% to 18.0%, and Kendall τ against real vLLM fell from 1.00 to −0.07. Replay also predicted Evolved, the real winner, to be slower than Stock. We have added the comparison with engine-based simulators to §2, the porting study to Appendix H, and the replay experiment to Appendix K (Table 20).

Thank you for the detailed rebuttal and additional experiments. The clarification of the policy interface and the new experiments address several of my original concerns, particularly regarding support for dynamic KV management and implementation effort.

However, I remain concerned about the generality of the proposed abstraction beyond its predefined operations and its advantages over simulators that directly reuse serving engines. I acknowledge the improvements and will raise my score from 1 to 2.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer 1oUh,

Thank you for reading our response and for raising your score. We briefly address the two remaining points.

Generality beyond the predefined operations. The operations are engine-level primitives: routing, ordering, admitting, and preempting calls, and protecting, releasing, discarding, offloading, restoring, prefetching, or migrating KV caches. Policies decide when and how to apply them, so new decision rules over these operations need no simulator changes; every optimization we evaluated is expressed this way. An optimization that needs additional engine behavior, such as a new transfer path, requires extending the engine model; that capability then becomes available to every policy through the interface. §3.4 states this boundary.

Advantages over engine-based simulators. The two approaches are complementary. Engine-based simulation directly reuses optimizations implemented in its target engine. AgentServeSim provides a common engine model for comparing and combining independently developed policies from different engines and engine versions. It also supports multi-instance routing and KV cache migration between instances, and preserves how each policy changes subsequent call arrivals and KV availability across tool waits.

We are happy to clarify anything further before the discussion period ends.

Best regards, The Authors

Reviewer_ay87OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. AgentServeSim extends LLMServingSim 2.0 with a program-level execution model tracking cross-call history, cache lifecycles, and resource constraints.
  2. It provides unified interfaces for cache management, scheduling, and routing, simulating how policy decisions affect subsequent calls.
  3. The authors validate prediction accuracy and policy rankings against real serving, then use OpenEvolve to search for policies and verify their performance gains.
Critical strengths
  1. Coordinated cache residency, protection, shared accounting, and admission checks allow program-level policies to be compared under consistent resource constraints.
  2. Across four configurations with five arrival rates each, mean program completion time MAPE is 4.74%. Simulation identifies the real-system winner among six candidates in all three ranking settings.
  3. The discovered policy achieves 1.25–1.45× speedups over the strongest evaluated alternative in each configuration, demonstrating the workflow’s practical utility.
Critical weaknesses
  1. The experiments do not distinguish the value of integrating search from the added value of the new simulation semantics for policy evaluation and selection.
  2. The boundaries between new execution capabilities and search integration remain unclear relative to Frontier, Vidur-Agent, and Glia [1–3].
  3. Search and ranking use a single instance; two-instance measurements are lightly loaded, and the four-instance routing comparison is simulation-only.

[1] Yicheng Feng et al. “Frontier: Towards Comprehensive and Accurate LLM Inference Simulation.” arXiv:2605.21312, 2026.

[2] Donghwan Kim et al. “Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulation.” arXiv:2606.01725, 2026.

[3] Pouya Hamadanian et al. “Glia: A Human-Inspired AI for Automated Systems Design and Optimization.” arXiv:2510.27176, 2025, revised 2026.

Questions for authors
  1. Which controlled comparisons distinguish the benefits of integrating search from those of the new program-level simulation semantics for policy evaluation and selection?
  2. Why are Frontier’s program state and history-aware scheduling classified as only partially supported?
  3. How severe was KV-cache pressure in the two-instance SAGA experiments?
6 comments

Since the authors didn't reply to my concerns, I'll keep my score.

AuthorsOpenReview ↗

Response to Reviewer ay87(1/3)

Thank you for the constructive review. The new results separate policy-evaluation fidelity from search outcomes and extend validation beyond one instance. On two instances, the policy found by a new two-instance search (Evolved-2) is 2.16× faster than Stock and 1.13× faster than Evolved on real vLLM; on four instances, the simulator reproduces the routing-policy order with mean-JCT errors within 3.3%. We also correct the description of Frontier's supported capabilities.

W1 and Q1. Which controlled comparisons separate the value of the execution semantics from the value of the search?

We ran a fixed-policy mechanism ablation and a matched-budget search comparison. The fixed-policy ablation shows that the semantics improve evaluation and ranking independently of search; in the search comparison, the full simulator finds a policy 1.42× faster than its seed on real vLLM, while without the semantics no proposal improves its seed by more than 1%.

First, the paper's(§5.3) six ranked policies run with identical policy code on real vLLM and in the simulator (one B200, Llama-3.1-70B, 50 SWE-Bench programs at 0.10 programs/s). With all three mechanisms off, the Seed simulation does not complete, so that row covers five policies.

SimulatorMean absolute JCT error over policiesKendall τ vs. real
Full2.9%1.00
Shared-prefix accounting off8.9%0.73
Admission-time revalidation off20.8%0.33
Copy waits off2.9%1.00
All three off (five policies)26.2%0.00

None of these six policies moves the KV cache to host memory, so turning off copy waits changes nothing here; for Autellix MLFQ, which does, it changes the error from −1.0% to −19.1% (our reply to S4WM, W2/Q2).

Second, the same search was run twice from Continuum with the same LLMs, budget, and 20 issue-disjoint SWE-LB search programs, changing only the evaluator: the full simulator, or the simulator with the three mechanisms off. Each search scores 51 proposals.

Search outcomeFull simulatorWithout the semantics
Proposals with the same score as the seed529
Proposals better than the seed by more than 1%310
Best proposal vs. the seed, simulated (search programs)1.18×1.00×
Best proposal vs. the seed, real (B200, 50 SWE-Bench test programs at 0.06 programs/s)1.42×1.00×

Third, with both simulators scoring the same 36 distinct policies the two searches proposed (excluding repeats and seed), the full simulator predicts the policy the search selected (1.42× faster than Continuum on real vLLM) to be the fastest of the 36, while the simulator without the semantics predicts it to be slower than Continuum (0.97×). The difference therefore comes from the simulator, not from the search's path.

We have added the fixed-policy ablation (Table 6) and the search comparison to §5.4, and the search details and the scoring of the 36 shared policies to Appendix B (Table 10).

AuthorsOpenReview ↗

Response to Reviewer ay87(2/3)

W2 and Q2. Why were Frontier's program state and history-aware scheduling marked as partial, and what is new beyond these simulators and LLM-driven search?

Frontier supports multi-round sessions with tool delays and history-aware scheduling; Table 1 now marks them as supported. Our contribution is a policy interface that controls serving decisions across a program's calls and tool waits, with the same policy code running in simulation and on real vLLM. LLM-driven search is an application of that interface.

Frontier (Feng et al., 2026) uses session history in its scheduler, and Vidur-Agent (Kim et al., 2026) supports four queue orders and load-aware routing. In both, a program's KV cache is reused between calls only through prefix-cache hits. AgentServeSim exposes program and instance state to policies that jointly control routing, scheduling, admission, preemption, and KV cache management. The simulator tracks how these choices affect memory, KV transfers, and when each program's next call arrives. On the search side, Glia (Hamadanian et al., 2025) uses LLM-driven search to design routing, scheduling, and autoscaling for request-level workloads, with no agent programs or tool waits. Our search targets multi-call agent programs, whose serving decisions change later arrivals and cache availability across tool waits. The table compares the four systems as each paper describes them.

FrontierVidur-AgentGliaAgentServeSim
Program-aware policy controlBuilt-in history-aware schedulerBuilt-in queue orders and load-aware routingRequest-level routing, scheduling, autoscalingRouting, scheduling, admission, preemption, and KV cache (retention, offload, prefetch, migration), at every engine step
Program-level policy interface shared with real serving✗✗✗✓
LLM-driven policy search✗✗✓ (request-level)✓ (agent programs)
Validation on real agent servingSimulation onlyOne trace, aggregate latencyNot reportedPer-policy error and ranking

Our abstraction enables developers to compare interacting agent-serving policies under shared execution semantics and validate the simulator's policy selection on real vLLM. The search relies on these semantics: scored without them, the same search finds no improvement (W1), so the contribution goes beyond adding a search loop.

We have corrected Table 1, updated the descriptions of Frontier, Vidur-Agent, and Glia in §2, and described the shared policy interface in §3.

Reviewer_S4WMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

AgentServeSim adds program-level execution and persistent cache state to an LLM-serving simulation backend. It releases successor calls from simulated completions and recorded tool waits, tracks shared-prefix ownership and protection, and rechecks cache availability and capacity at admission. Across 20 hardware comparisons it reports 4.74% MAPE in mean program JCT. A simulator-guided search over KV-management and scheduling code produces a policy that is also the best of six evaluated policies in three real-serving settings, with 1.64–2.04× speedup over FCFS/LRU and 1.25–1.45× over the strongest evaluated alternative.

Critical strengths
  1. The execution semantics address consequential simulator errors. Sections 3.2–3.4 explain why fixed successor arrivals, duplicated prefix accounting, premature transfer availability, and stale routing assumptions can change policy conclusions. Separating future trace data from observations available to policies is particularly valuable for avoiding unrealistic policies during automated search.

  2. The simulator is validated on the decision it is intended to support. Beyond aggregate latency error, Table 10 reports policy-level hardware results and ranking disagreements. The selected policy improves real mean JCT by substantial margins in all three settings; for example, RTX SWE-Bench JCT falls from 4,346.924 seconds for Seed to 2,991.206 seconds for Evolved. This is stronger evidence of utility than simulation-only speedups.

Critical weaknesses
  1. The transfer and selection evidence is narrower than a fully held-out policy-search evaluation. Appendix B explicitly states that development and test programs are not established to be disjoint. Only one 50-iteration search and three six-policy ranking settings are reported, without uncertainty across workload samples, arrival seeds, or repeated searches. The hardware gains are convincing for those settings, but they do not yet establish how reliably the simulator selects policies on genuinely unseen programs or among closely matched search candidates.

  2. The component analysis changes policy semantics as well as simulator mechanisms. In Table 5, disabling retention or replacing program-aware ordering with FCFS is compared against an unchanged real Continuum reference; Appendix D likewise uses the backend's default policies. These are useful mismatch controls, but they do not isolate the added fidelity from shared-prefix accounting, transfer readiness, or admission-time revalidation under matched policy logic. The B200 error improving from 4.50% to 1.60% without the KV component illustrates why aggregate JCT alone cannot resolve that attribution.

Questions for authors
  1. What is the task/repository overlap between the 20-program search subset and the SWE-Bench transfer set? Can the policy's advantage and ranking agreement be shown on an explicitly disjoint split with paired repeats or independent arrival seeds?

  2. Can a matched-policy comparison disable one execution-model mechanism at a time, or report targeted traces showing its effect on cache occupancy, recomputation, and admission timing? This would clarify which new semantics drive the fidelity improvement.

  3. For real replay, how closely do the implementations of Autellix, Continuum, and SAGA match their published decision rules and tuning? Please identify any adaptations that could materially affect Table 10.

2 comments
AuthorsOpenReview ↗

Response to Reviewer S4WM(1/2)

Thank you for the thorough review. On the reported disjoint evaluations, AgentServeSim selects the same winner as real serving, and Evolved retains a 1.22–1.47× advantage over the strongest alternative across three BFCL arrival seeds. The matched-policy ablations identify which execution mechanisms preserve fidelity, and we clarify baseline fidelity.

W1/Q1. What is the overlap between search and test programs, and do the policy's advantage and ranking agreement hold on disjoint programs and independent arrival seeds?

We audited the overlap, evaluated the Django subset and BFCL, replayed BFCL with two more arrival seeds, and ran a second search from a different initial policy, three times. On every disjoint evaluation and arrival seed, Evolved keeps its real-vLLM advantage and the simulator picks the real winner.

(a) Overlap (Appendix A). Eight of the 50 SWE-Bench test programs share an issue ID with the 20 search programs. They are separate agent runs, with call counts differing by 8–21, final prompts 1.2–2.4 times smaller, and total tool wait time differing by up to 50×. All 20 search programs are from astropy; the test set contains 10 astropy and 40 django programs. BFCL v4 shares no task or repository with the search set: its 150 web-search programs have a median of 8 calls, versus 44 in the search set, and a median tool wait time nine times longer.

(b) Disjoint programs (Appendix G, Table 14). The Django rows are computed over the 40 django programs of the 50-program runs; BFCL is disjoint by construction.

Disjoint splitKendall τ, real vs. simWinner (real / sim)Evolved vs. Stock, realEvolved vs. best alternative, real
RTX SWE-Bench, django programs only (40)1.000Evolved / Evolved1.80×1.53× (Seed)
B200 SWE-Bench, django programs only (40)0.733Evolved / Evolved1.66×1.43× (Continuum)
RTX BFCL (150)0.867Evolved / Evolved2.04×1.25× (Continuum)

On all three disjoint splits, the simulator picks the real winner, and Evolved is 1.25–1.53× faster than the best alternative on real vLLM.

(c) Independent arrival seeds (Appendix G, Table 15). We repeated the BFCL setting (2× RTX PRO 6000, Llama-3.1-70B, 150 programs at 0.10 programs/s) under two new arrival seeds.

Arrival orderKendall τ, real vs. simWinner (real / sim)Evolved vs. Stock, realEvolved vs. best alternative, real
Original0.867Evolved / Evolved2.04×1.25× (Continuum)
Order 10.867Evolved / Evolved1.89×1.47× (Seed)
Order 20.867Evolved / Evolved2.14×1.22× (Continuum)

Under every arrival order, the simulator ranks the policies equally well (τ = 0.867) and picks the real winner, and Evolved stays 1.22–1.47× ahead of the best alternative.

(d) A second search configuration (Appendix B, Table 9) and repeated searches. We ran a second search with the same budget and LLMs as the paper, started from Continuum instead of the paper's search seed, on 20 SWE-LB programs that share no issue with the SWE-Bench test programs.

SettingBest policy vs. its seed (Continuum), realBest policy vs. Stock, real
RTX SWE-Bench2.36×2.82×
B200 SWE-Bench1.42×1.73×
RTX BFCL1.11×1.60×

Started from a different policy and trained on different programs, the search again finds a policy faster than its seed in all three settings (1.11–2.36×). We repeated this search with two more random seeds. On B200 SWE-Bench, one found a policy 1.42× faster than Continuum on real vLLM; the other found only small gains in simulation (1.02×) and did not improve on Continuum on real vLLM (0.93×).

The Django and BFCL results show that the benefit is not confined to the search programs, and BFCL additionally tests much shorter call sequences and longer tool waits. The second search and its repeats show that the search can find a strong policy from a different starting policy, training set, and random seed.

AuthorsOpenReview ↗

Response to Reviewer S4WM(2/2)

W2/Q2. Can a matched-policy comparison disable one execution mechanism at a time and show, through traces, how each affects cache use, recomputation, and admission?

We ran identical policy code on real vLLM and in the simulator, disabling one execution mechanism at a time. Under matched policy logic, shared-prefix accounting and admission-time revalidation materially affect JCT. Copy waits matter for policies that move KV to host memory. For the four policies below, the full model's absolute mean-JCT error is at most 2.3%; individual ablations increase error to as much as 43.8%.

The setting is one B200, Llama-3.1-70B, 50 SWE-Bench programs at 0.10 programs/s, and a 65,536-token KV pool. Entries are signed relative errors in mean JCT versus the same policy on real vLLM.

PolicyFull modelShared-prefix accounting offAdmission-time revalidation offCopy waits offAll three off
Stock+1.1%+1.5%−27.2%+1.1%−26.8%
Continuum−0.8%+43.8%−11.7%−0.8%+32.7%
SAGA+2.3%−0.3%−31.1%+2.3%−30.8%
Autellix MLFQ−1.0%−0.4%−33.4%−19.1%−41.7%

The event traces explain the direction of the errors:

  • Shared-prefix accounting: without it, a protected prefix and the call reusing it are charged twice. Continuum's preemptions rise from 51 to 744, and recomputed prefill grows from 7.0 M to 10.1 M tokens (real: 6.5 M).
  • Revalidation: without it, calls keep credit for KV cache evicted while they waited. Under SAGA, 1,827 calls skip 12.8 M tokens of recomputation that real serving performs.
  • Copy waits: MLFQ moves about 1.7 TB to and from host memory. Making those copies free understates JCT by 19.1%.

We have added the matched-policy ablation to §5.4 (Table 6).

Q3. How closely the baselines follow their published designs.

Appendix I (Baseline Policies) of the revised paper describes each baseline; Table 10 is table 13 in revision:

  • Continuum: we follow its released code, which uses a 2 s TTL: when a call finishes, its KV cache is protected for 2 s if the tool's mean duration is at most 2 s (or the tool is new), and programs whose KV cache is protected are scheduled first.
  • Autellix: Table 10 used its program-level priority (PLAS) without MLFQ swapping. The full MLFQ with swapping is in the ablation above, where the simulator is within 1.0% of real. In the Table 10 settings it is slower than Stock in simulation (0.70× to 0.80×).
  • SAGA: we follow its paper: per-tool TTL retention, workflow-aware eviction, prefetch, and work stealing across instances. Where the paper leaves a setting open, we use 16 calls in flight per instance and steal only from an instance with at least twice the load.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.