acceptodds
Under review as a conference paper at ICLR 2027

AgentSimArena: Benchmarking LLM Serving Simulators for Agent Workloads

Abstract

Serving simulators can reduce the cost of exploring LLM-agent deployments, but inaccurate predictions can lead to poor deployment choices. Evaluating these simulators is difficult because each predicted call completion determines when sub- sequent work arrives, changing future contention. We introduce AgentSimArena, a benchmark that preserves this feedback while comparing simulator predictions with real executions under matched workload and deployment constraints. Across six simulators, eight deployments, and five arrival rates, we evaluate 240 predic- tions against 40 vLLM reference runs. LLMServingSim 2.0 achieves the lowest mean absolute relative error in mean program completion time, at 7.1%; DynoSim achieves 10.6%, with a median simulation runtime of 6.86 s versus 1,224.26 s for LLMServingSim. No simulator remains within 10% error across all settings. In one observed deployment comparison, following a simulator’s recommendation selects an alternative with 6.07× the measured mean program completion time of the other deployment. Arena documents integration extensions, modeling differences, and available preparation-cost evidence, providing a basis for workload-specific simulator selection.

Primary area:
datasets and benchmarks
Keywords:
LLM serving, agentic workloads, simulation benchmarking, simulator fidelity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kWS5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AgentSimArena, a benchmark for evaluating the accuracy and efficiency of existing LLM serving simulators under agent workloads. The benchmark replays fixed agent execution traces while preserving dependencies between consecutive LLM calls, and compares simulator predictions against real vLLM executions under matched deployment configurations.

Update:

After reading the authors' rebuttal, I tend to keep my current score.

Critical strengths
  1. A standardized comparison of simulators against real serving systems can provide useful guidance for practitioners
  2. The paper establishes a common evaluation protocol across six existing simulators, aligning workload inputs, deployment constraints, and performance metrics
Critical weaknesses

The paper mainly observes that simulators exhibit different prediction errors across hardware and workload configurations, and that accuracy comes with varying computational costs. I feel this is largely expected given their different performance models, profiling data, and supported mechanisms. The paper primarily quantifies these differences without establishing sufficiently novel or generalizable insights.

Although the paper reports substantial differences in prediction accuracy, it provides limited causal analysis of the underlying reasons.

Despite reporting 240 predictions, the evaluation covers only eight underlying deployment configurations, five arrival rates, and a single real serving engine. The results also lack sufficient repeated measurements and uncertainty estimates.

Questions for authors

Check weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer kWS5 (1/2)

We thank the reviewer for the review. We traced the errors of all six simulators to three general causes, now in Section 6: missing prompt reuse, memory pressure and step timing. Usual validations miss the first two. A repeat of all 40 real runs leaves the ranking unchanged.

W1: General insights from the error analysis.

We agree that different models give different errors. Our new analysis shows which modeling choices cause them on agent workloads. The pattern holds across all six simulators (table below, 40 settings).

  1. Missing prompt reuse causes the largest errors. An agent call re-sends almost all of its earlier context. Real vLLM serves 96–98% of each SWE prompt from its cache when memory allows. The two simulators without prefix caching have mean errors of 97% and 280%, and their error grows with real reuse. The four with prefix caching have 7–39%.
  2. Memory pressure turns small timing errors into large ones. Above some load, the real cache evicts context before the next call can reuse it, and queues build. A slightly fast simulator never reaches that load and predicts no queue. Frontier on Llama-8B (RTX PRO 6000) is 36% short on decode time at the lowest load. At higher loads its JCT error is −86% to −88%.
  3. Once reuse is modeled, step timing causes the rest. LLMServingSim's prompt processing is too fast. TokenSim scales the attention time of a call's new tokens by the work of attending to the whole cached prompt. So calls with few new tokens are the slowest. On C0 at 0.10 programs/s, its median first step is 204 ms with 32–63 new tokens and under 50 ms with 256 or more.

Usual validations miss these findings. Independent requests share at most a common prefix, such as a system prompt, not a program's earlier calls, so they cannot show finding 1. Replay at recorded timestamps hides finding 2: Frontier's mean error drops from 34.4% to 8.2% with its model unchanged. Arena shows both because each call waits for the previous one, in simulation and in real serving. For simulator developers, accurate step timing is necessary but not enough on agent workloads: APEX's decode-only steps match real vLLM (16.4 vs 15.8 ms), yet without prefix caching its mean error is 280%.

SimulatorPrefix cachingMean errorError vs real reuse (Spearman ρ)
LLMServingSim 2.0yes7.1%0.38
DynoSimyes10.6%−0.10
Frontieryes33.8%0.38
TokenSimyes39.4%0.37
SplitwiseSimno97.0%0.62
APEXno279.7%0.75

ρ is the rank correlation between a simulator's error and real prompt reuse. Near 1, the error grows with reuse. Near 0, it does not depend on it.

AuthorsOpenReview ↗

Response to Reviewer kWS5 (2/2)

W2: Causal analysis of the accuracy differences.

We now trace the errors to their causes (Section 6, Appendix H.6). For all 240 predictions, we split each program's time into time to first token (queueing plus prompt processing) and decode time. For the worst cases, we logged every simulated step.

  • APEX on Llama-70B, B200, TP2 (+995% to +1,180%): real vLLM takes 98% of each prompt from its cache. APEX has no prefix caching or chunked prefill. It recomputes the whole prompt (median 13,438 tokens) at every call, in one step that blocks all decoding. Its decode-only steps are accurate (16.4 ms vs 15.8 ms), but prompt steps take 86% of its simulated time.
  • LLMServingSim on Llama-70B, B200, two instances (−19% to −24%): the real cache is small (83,312 tokens per instance). It evicts context before the next call can reuse it, so calls queue for minutes. LLMServingSim's prompt processing is too short in all eight deployments (0.36–0.79× real). Here prompt work is large, so shorter prompts shorten the queue: 477 s vs 575 s per program at the highest load.
  • Over the 40 settings, APEX and TokenSim are too slow in both phases. Their time to first token is 2.13× and 1.38× real, and their decode time 2.29× and 1.26×. LLMServingSim is at 0.96× in both. Its time to first token includes queueing, so it can look close to real while prompt processing alone is too short.

W3: Deployments, arrival rates, engine and repeated runs.

  • Repeats (Section 5.2, Appendix H.2): we repeated all 40 real runs. Mean JCT changed by 2.0% on average and by at most 9.2%. 35 of 40 settings are within 5%. Each simulator's error moves by at most 2 points, and the order is unchanged. In the table below, run 1 is the paper's real runs and run 2 the repeat.
  • Deployments: the eight cover two GPU types, three models (dense and mixture-of-experts), tensor parallelism and replication. The most accurate simulator changes between them: DynoSim in four, LLMServingSim in two, TokenSim in two. Adding a deployment needs a configuration file, real runs, and profiles for GPUs a simulator does not yet cover.
  • Arrival rates: the five rates go from light load to loads where, in memory-limited deployments, the real cache can no longer hold each program's context (finding 2).
  • One engine: all real runs use vLLM 0.19, and Section 7 now states this as a limitation. We leave a comparison against a second serving engine, such as SGLang, to future work.
SimulatorMean error, run 1Mean error, run 2
LLMServingSim 2.07.1%7.0%
DynoSim10.6%9.8%
Frontier33.8%34.8%
TokenSim39.4%39.9%
SplitwiseSim97.0%97.5%
APEX279.7%281.7%

I appreciate your thorough response. The new error analyses help clarify why the simulators failed, and running the experiments repeatedly on the real system makes readers confident in the overall findings. My remaining concern is whether these findings provide sufficiently novel and generalizable insights beyond diagnosing the evaluated simulators under the tested configurations, which I believe is only partially addressed.

Reviewer_GTgFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AgentSimArena evaluates six LLM-serving simulators against vLLM on agent workloads. It preserves completion-dependent execution: each branch releases subsequent calls according to its own completion times and recorded tool delays. The evaluation covers eight deployments and five arrival rates, comparing prediction accuracy, simulation runtime, and profiling costs. The paper addresses a relevant systems problem, but important questions remain about experimental equivalence and differentiation from prior work.

Critical strengths

1. Relevant problem and a sensible execution protocol.
Errors in serving simulation can lead to poor deployment decisions. Preserving the feedback between call completion and subsequent demand is appropriate for sequential agent workloads. The inspected APEX and TokenSim integrations implement this dependency explicitly, providing concrete support for the proposed protocol.

2. Useful comparison of accuracy and practical cost.
The evaluation shows that aggregate accuracy can conceal substantial differences across deployments and loads. Reporting accuracy alongside runtime and preparation costs makes the benchmark useful for selecting a simulator. The analysis of deployment choices and load-response differences is more informative than a single average-error ranking.

3. Substantial integration work and explicit checks.
The inspected adapters check completion coverage, dependency timing, token counts, and KV-capacity translation. The reporting script also specifies how runs are selected, rather than leaving this entirely implicit. These details strengthen the engineering contribution and make several assumptions inspectable.

Critical weaknesses

1. The distinction from the closest prior work needs to be clearer.
AgentServeSim already studies paired real/simulated execution with completion-dependent agent calls, while LLMServingSim 2.0 includes cross-simulator validation. The main contribution here is therefore the common benchmark, integration audit, and broader comparative evidence. This can merit publication, but the paper should establish more precisely what new methodology or conclusions arise beyond these existing efforts. An ablation against fixed-timestamp replay would help quantify the benefit of the central protocol.

2. The claim of matched experimental conditions is insufficiently validated.
TokenSim’s adapter explicitly states that the specified sequence and batch-token limits are not enforced. APEX retains an upstream 65,536-token batch cap rather than the reference’s 16,384-token limit. Consequently, some error differences may reflect different effective serving policies as well as timing predictors.

Separately, LLMServingSim constructs cache hashes from recorded input and output token IDs, while Appendix C.1 permits real inference to produce different output text. The paper does not establish how equivalent cache-reuse opportunities are maintained. This is an unresolved validation question, not a demonstrated implementation bug, but it could affect the amount of prefill work being compared.

3. Evaluation scope and uncertainty limit the broader conclusions.
The experiments use prescribed sequential call chains with fixed prompts, output lengths, and tool delays. This is a valid controlled serving workload, but it does not establish generality to branching programs, parallel subagents, or tool delays that change with system load. Furthermore, the reported comparisons retain one selected observation per setting, without repeated-run uncertainty. Stability checks are particularly important for real latency measurements and runtime-based Pareto counts.

Questions for authors
  1. Does real replay force the recorded output token IDs? If not, how much does the resulting token-content difference affect prefix-cache reuse?
  2. Can the unmatched batching limits be aligned, or can their contribution to the reported errors be quantified?
  3. Can the complete raw results, workload/configuration manifest, reference replay driver, and missing external dependencies be released?
4 comments
AuthorsOpenReview ↗

Response to Reviewer GTgF (1/3)

We thank the reviewer for the careful review and the concrete suggestions. With fixed timestamps, the three most accurate simulators look 2–7× more accurate with their models unchanged. Aligning the batching limits moves mean errors by at most 5.1 points, and forcing the recorded outputs by less than 1 point. A repeat of all 40 real runs leaves the ranking unchanged. The raw results, replay driver and dependencies are released.

W1: Distinction from AgentServeSim and LLMServingSim 2.0, and the fixed-timestamp ablation.

  • Distinction: AgentServeSim pairs real and simulated runs for one simulator. LLMServingSim 2.0 validates its own predictions and compares other simulators on its own setup. Arena fixes programs, real runs, limits and completion-dependent release for six simulators. It checks every run, and our audit found three simulators that ignored limits they were given (W2 (a)). It compares internal state (cache reuse, queues, step times) with the real engine.
  • Results neither work reports (Section 6):
    • Missing prompt reuse causes the largest errors. Error grows with real reuse for APEX and SplitwiseSim (Spearman ρ = 0.75 and 0.62), but not for DynoSim (ρ = −0.10).
    • Memory pressure turns small timing errors into large ones.
  • Ablation: the table below gives mean error over 16 settings, 8 deployments at 0.02 and 0.10 programs/s (Section 6, Table 5).
SimulatorCompletion-dependent: JCT errorFixed timestamps: JCT error
LLMServingSim 2.06.8%1.0%
DynoSim10.8%4.8%
Frontier34.4%8.2%
TokenSim39.4%179.2%
SplitwiseSim84.6%74.7%
APEX275.8%290.3%

With fixed timestamps, each call starts when it did in the real run, so JCT mostly follows the real timeline (Appendix H.7).

  • The three most accurate simulators look 2–7× more accurate, although their models are unchanged.
  • Simulators slower than real vLLM (TokenSim, APEX) still get calls at the real pace. Their queues grow, and their errors rise.

W2 (a) and Q2: Aligned batching limits and their effect.

You are right about TokenSim. We aligned the limits wherever a simulator can express them and reran every affected setting. Mean errors change by at most 5.1 points. The ranking does not change. The table below gives mean error over the 40 settings.

  • TokenSim now enforces the 128-sequence cap and a token budget per step. SplitwiseSim also lacked a token budget. It now has one.
  • APEX's 65,536-token cap is deliberate. Without chunked prefill, a whole prompt must fit in one step, and vLLM itself requires a budget of at least the context length when chunking is off (Table 2, Section 4.4). For Mixtral we lowered it to its 32,768-token context.
  • The remaining difference is that three simulators cannot chunk prefill. In APEX, steps that contain a whole prompt take 81–89% of simulated time in the 70B deployments.
SimulatorBeforeAfterWhat changedSettings affected
TokenSim39.4%39.5%128-sequence cap and token budget per step16 of 40 (at most 3.5 points)
SplitwiseSim97.0%91.9%token budget per step30 of 40
APEX279.7%282.7%token budget 65,536 → 32,768 for Mixtral10 of 40
LLMServingSim 2.0, DynoSim, Frontier7.1%, 10.6%, 33.8%unchangednothing (already enforced)0
AuthorsOpenReview ↗

Response to Reviewer GTgF (2/3)

W2 (b) and Q1: Recorded outputs and cache reuse.

The paper's real runs do not force outputs. Both sides still have the same reuse opportunity: the earlier prompts. Forcing the outputs changes neither real reuse nor simulator error much (Appendix H.5).

  • Paper runs: the next prompt carries the recorded output, not vLLM's own. Real vLLM computes it as new tokens once, then reuses it like any prompt text. The simulators reuse only earlier prompts. With an unlimited cache, earlier prompts allow 48.5% reuse on BFCL and 96.3% on SWE-LB. Real vLLM reaches 48.0% and 96%.
  • Forced runs: in 16 settings (each deployment at 0.02 and 0.10 programs/s), a logits processor makes vLLM produce each call's recorded output. This covers every call whose output the next prompt contains verbatim (59.5–79.6% of calls). LLMServingSim hashes the recorded outputs, and DynoSim gets them as its generated tokens.
    • Real runs: the cached share of prompt tokens changes by at most 1.0 point. Mean JCT changes by at most 5.9%, except in C0 (+9.6% and +19.0%), where each output token takes longer to generate.
    • Simulators: mean error over the 16 settings changes by less than 1 point (table below). The paper's 7.1% and 10.6% are over 40 settings against the unforced runs.
SimulatorEarlier prompts onlyAlso recorded outputsLargest change in one setting
LLMServingSim 2.09.4%10.2%6.8 points
DynoSim12.4%12.0%12.2 points

W3 (a): Branching programs, parallel subagents and load-dependent tool delays.

The paper's programs are sequential call chains with recorded tool delays, and Section 7 now states this scope. The release rule extends directly to branching programs and parallel subagents: a call starts when all its parents finish. We ran one such setting: 46 programs with parallel subagents from the FreeInference trace, arriving at 0.1 programs/s, on Llama-3.1-8B with one B200. Tool waits are capped at 600 s, and real vLLM is forced to produce the recorded outputs, as in W2 (b). Real vLLM's mean JCT is 7,573 s. TokenSim predicts +17%, DynoSim −20%, LLMServingSim −22%, SplitwiseSim −34% and APEX +44%. Frontier does not support parallel subagents. Each tool wait uses its recorded delay. Modeling tool delays that grow with system load, such as a tool waiting on a busy shared service, is future work.

AuthorsOpenReview ↗

Response to Reviewer GTgF (3/3)

W3 (b): Stability of real runs and Pareto counts.

  • Real latencies: we repeated all 40 real runs. Mean JCT changed by 2.0% on average and by at most 9.2% (Section 5.2, Appendix H.2).
  • Simulator errors: against the repeat, each mean error moves by at most 2 points, and the order is unchanged.
  • Pareto counts: a simulator counts in a setting if no other simulator is both more accurate and faster. Simulation time varies between reruns (0.34–2.34×). We recounted against the repeat runs, with runtimes within 2× treated as ties (Appendix H.3). The table below gives the counts out of 40 settings. DynoSim keeps 36 of 39 and LLMServingSim 14 of 15, so the RQ3 result holds. SplitwiseSim falls from 28 to 5, because in 24 settings DynoSim is more accurate and only 1.0–2.3× slower. Section 5.3 now reports the counts with this tolerance.
SimulatorPaperAgainst repeat real runs2× runtime toleranceBoth
DynoSim39393636
LLMServingSim 2.015141514
SplitwiseSim282855
TokenSim8383
Frontier5401
APEX3333

Q3: Released results, manifest, replay driver and dependencies.

The anonymous repository linked in the paper now contains the raw results of the 40 real runs, their repeat, the 240 simulator runs and the reruns in this response. It also contains a manifest of the 40 settings with all 15 workloads, the reference replay driver that produced the real runs, pinned dependencies for each simulator, and the adapters with the batching limits of W2 (a).

Thank you for the extensive additional experiments. However, my concerns about novelty remain unresolved, so I will maintain my original score.

Reviewer_s7PSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents AgentSimArena, a benchmark for evaluating LLM serving simulators on agent workloads. It replays fixed agent traces (SWE-LB, SWE-bench, BFCL) in both simulators and a reference vLLM deployment, releasing each call after its predecessor completes plus a tool delay. Simulators are configured with KV capacity measured from the real engine, and several receive Arena-specific integration extensions. Across six simulators, eight deployments, and five arrival rates (240 predictions vs. 40 real runs), LLMServingSim 2.0 achieves the lowest mean JCT error (7.1%), DynoSim is close (10.6%) and ~178x faster, and no simulator stays within 10% error in all settings.

Critical strengths

The paper tackles a realistic and timely problem. Whether LLM simulators remain accurate under agentic multi-call, completion-dependent execution is a practically important question.

The empirical evaluation is reasonably broad (six simulators, two GPU types, three models, TP and replicated deployments, five arrival rates). This is a useful data point for practitioners currently choosing among these specific simulators.

The paper reports accuracy together with simulator runtime and profiling cost, and shows that aggregate rankings do not transfer across deployments.

Critical weaknesses

Limited novelty in the benchmark construction. Completion-dependent call release is the natural way to replay agent programs and is already natively supported by DynoSim, LLMServingSim 2.0, and Frontier. The workloads are existing traces, the metric is mean JCT error, and validating simulators against real measurements is what each simulator paper already does. The paper introduces no new workload, metric, or methodology. The contribution reduces to running existing simulators and reporting their errors.

Matched workloads are a minimum requirement, and aligning with real measurements is unmotivated. Using the same trace, release semantics, and deployment during comparison is a prerequisite for any valid comparison, not a contribution. It is also unclear why simulators should be calibrated to a real JCT point. This could potentially hide modeling errors the benchmark should expose. Whether other vLLM settings (max sequences, batched tokens, chunked prefill, prefix caching) are matched in each simulator is also unclear.

Integration adaptations are unclear and confound results. Four of six simulators are modified by Arena, each described in about one sentence and validated only by "small checks." It does not explain what the integration aims to preserve. Each setting is also a single run with no variance reported.

No insight into why simulators succeed or fail. The results are a leaderboard of error numbers. Large failures are not diagnosed, and Section 6 offers only hypotheses. The paper does not tell readers what current simulators miss for agent workloads. In evaluation, RQ1 is a feature survey unrelated to Arena's design.

Poor presentation. The writing is dense and abstract, key terms ("program", "cell", "branch", "extension") are used before being defined, many sentences are hard to parse, and much of the method is deferred to the appendix.

Questions for authors

Beyond running existing simulators under a completion-dependent replay, what is the methodological contribution of Arena? What could a simulator developer or user learn from Arena that they could not obtain from a standard validation run against their own target deployment?

How do the results change if each simulator derives capacity from its own memory model using only the model, GPU, and engine configuration?

Are the maximum sequence count, batched-token limit, chunked prefill, prefix caching, and scheduling policy matched in every simulator, and where they cannot be, how is the mismatch accounted for?

Can the authors provide root-cause analysis for the simulator errors (e.g. APEX in C1, LLMServingSim 2.0 in C3), for example by decomposing JCT error into queueing, prefill, decode, and cache-reuse components? What general lessons about simulating agent workloads follow from these analyses?

Each setting is reported as "one selected run." How were runs selected? what is the run-to-run variance of the real vLLM reference?

3 comments
AuthorsOpenReview ↗

Response to Reviewer s7PS (1/3)

We thank the reviewer for the detailed review. We added a root-cause analysis of all 240 predictions: missing prompt reuse causes the largest errors, and memory pressure turns small timing errors into large ones. Releasing calls at their recorded times instead changes every simulator's error. No simulator is calibrated to a real JCT, and a repeat of all 40 real runs leaves the ranking unchanged.

W1 and Q1: Arena's contribution beyond completion-dependent replay.

Arena contributes a protocol and following findings.

  • Protocol: everything is fixed except the simulator. Programs, limits and KV capacity are the same. The release rule is enforced in simulation and in real serving. Every run is checked automatically, and our audit found three simulators that silently ignored limits they were given (Q3).
  • Finding 1: missing prompt reuse causes the largest errors, and memory pressure makes small timing errors large (W4). Validations with independent requests cannot show this.
  • Finding 2: accuracy does not carry over between deployments. The most accurate simulator is DynoSim in four deployments, LLMServingSim in two and TokenSim in two. Validating on one deployment picks the wrong simulator for at least half of the others.

The release rule is not new, but it must be enforced on both sides. The common alternative, release at recorded times, changes the results. The table below gives mean error over 16 settings, 8 deployments at 0.02 and 0.10 programs/s (Section 6, Table 5; Appendix H.7).

SimulatorCompletion-dependent: JCT errorRecorded times: JCT error
LLMServingSim 2.06.8%1.0%
DynoSim10.8%4.8%
Frontier34.4%8.2%
TokenSim39.4%179.2%
SplitwiseSim84.6%74.7%
APEX275.8%290.3%

With recorded times, JCT follows the real timeline, not the simulator.

  • The three most accurate simulators look better, although their models are unchanged.
  • Simulators slower than real vLLM (TokenSim, APEX) still get calls at the real pace. Their queues grow, and their errors rise.

W2 and Q2: Calibration and each simulator's own KV capacity.

No simulator is calibrated to a real JCT in the main comparison. We align only the KV-cache capacity, the number of tokens that fit in memory. This fixes an input, not an outcome. Each simulator still predicts every completion time with its own timing model (Section 3.1).

With each simulator's own capacity, mean errors change by at most 2 points. The order stays the same (Appendix H.4). The table below covers all 8 deployments at 0.10 programs/s.

SimulatorOwn capacity vs sharedMean error, shared capacityMean error, own capacity
LLMServingSim 2.0+2% to +20%7.2%8.5%
DynoSim−16% to 0%13.2%13.0%
Frontier+4% to +36%33.6%31.6%
TokenSim+12% to +76%55.9%54.9%
SplitwiseSim+15% to +91%110.2%110.5%
APEX+16% to +103%311.6%309.6%

Most simulators assume up to 2× more memory than vLLM has, because they leave out activation and runtime memory. Errors move only where memory is tight (C3, C5), in both directions, so the main results share one capacity.

Scaling to a real JCT appears only in a separate post-hoc test (Section 6, Appendix G). It separates constant-factor errors (DynoSim 11.3% → 5.6%) from errors that change with load (APEX keeps 53.7%).

AuthorsOpenReview ↗

Response to Reviewer s7PS (2/3)

W2 and Q3: Matched serving limits in every simulator.

Every simulator matches the limits it can express (Table 2, Section 4.4). All six use 128 sequences, FCFS and round-robin routing. SplitwiseSim, APEX and TokenSim cannot chunk prefill, so a whole prompt must fit in one step. Their token budget is therefore the context length, the value vLLM itself requires when chunking is off. SplitwiseSim and APEX also lack prefix caching.

Our audit found three gaps, and we fixed them: TokenSim ignored the 128-sequence cap, TokenSim and SplitwiseSim had no token budget, and APEX's budget was too high for Mixtral. Rerunning every affected setting (Appendix H.1) changes mean errors little: TokenSim 39.4% → 39.5%, SplitwiseSim 97.0% → 91.9%, APEX 279.7% → 282.7%. The ranking does not change. W4 analyses the cost of missing chunked prefill and prefix caching.

W3: What the integrations preserve, change and check.

  • Preserved: each simulator's own scheduler and timing model decide when calls run and finish.
  • Added: only what the protocol needs. This is release after the predecessor plus the tool delay (in SplitwiseSim, APEX and TokenSim), round-robin routing, the same KV capacity and vLLM's batching limits.
  • Missing from the submission: four fixes in Frontier and SplitwiseSim. The revised Table 7 lists every change, why it is needed and how it is checked. Section 3.4 states what the integration preserves.
  • Checks: every run is checked automatically and stops on failure. The checks cover call order and tool delays, token counts, KV capacity and instance placement.

W3 and Q5: Run selection and real run-to-run variance.

  • Selection: no run was chosen for its numbers. Each setting has exactly one run that completed. All other attempts failed at startup and produced no results.
  • Variance: we repeated all 40 real runs (Section 5.2, Appendix H.2). Mean JCT changed by 2.0% on average and by at most 9.2%.
  • Effect: each simulator's mean error moves by at most 2 points, and the order is unchanged. The most accurate simulator is the same in 31 of 40 settings.
AuthorsOpenReview ↗

Response to Reviewer s7PS (3/3)

W4 and Q4: Root causes of the errors and general lessons.

For all 240 predictions, we split each program's time into time to first token (queueing plus prompt processing) and decode time (Section 6, Appendix H.6). We relate each error to how much of each prompt real vLLM took from its cache. In the table below, times are simulated ÷ real per program over the 40 settings; above 1 means too slow.

SimulatorPrefix cachingMean errorTime to first tokenDecode timeError vs real reuse (Spearman ρ)
LLMServingSim 2.0yes7.1%0.960.960.38
DynoSimyes10.6%1.241.09−0.10
Frontieryes33.8%0.940.880.38
TokenSimyes39.4%1.381.260.37
SplitwiseSimno97.0%––0.62
APEXno279.7%2.132.290.75

ρ is the rank correlation between error and real reuse; near 1, the error grows with reuse. SplitwiseSim records only when calls complete, so its time cannot be split.

APEX in C1 (+995% to +1,180%). Real vLLM takes 98% of each prompt from its cache. APEX has no prefix caching or chunked prefill, so it recomputes the whole prompt (median 13,438 tokens) in one step that blocks all decoding. These steps take 86% of its simulated time, while its decode-only steps match real vLLM (16.4 vs 15.8 ms).

LLMServingSim in C3 (−19% to −24%). C3's small cache evicts context before the next call can reuse it, so prompt work is large and calls queue for minutes. LLMServingSim's prompt processing is too short in all eight deployments (0.36–0.79× real). In C3 this shortens the queue: 477 s vs 575 s per program at 0.10 programs/s (Appendix H.6). Where most of each prompt is reused (C1, C7), LLMServingSim stays within 5%.

General findings (Section 6, replacing the hypotheses):

  1. Missing prompt reuse causes the largest errors. Real vLLM serves 96–98% of each SWE prompt from its cache when memory allows. The two simulators without prefix caching have the largest errors, growing with real reuse.
  2. Memory pressure turns small timing errors into large ones. Above some load, the real cache evicts context and queues build. A slightly fast simulator never reaches that load. Frontier on C4 is 36% short on decode time at the lowest load and −86% to −88% in JCT at higher loads. This appears only when each call waits for the previous one.
  3. Once reuse is modeled, step timing causes the rest. LLMServingSim's prompt processing is too fast, and TokenSim overestimates calls with few new tokens on long cached prompts.

RQ1 now links to these results (Section 5.1): its capability table predicts that the two simulators without prefix caching have the largest errors. We also corrected DynoSim's prefix caching from partial to full. DynoSim does not reuse generated tokens, but neither does real vLLM in our replay (Appendix H.5).

W5: Presentation.

The revised paper defines every term at first use and replaces "cell" and "branch" with plain words. We simplified dense passages. The release rule moved from the appendix into Section 3.3, and Section 4.4 adds the serving-limits table.

Comments on the paper

2 comments

Dear Reviewers,

Please review the authors' rebuttal and participate in the discussion to address any remaining questions before the period closes on Nov 18 AoE.

Thank you!

AC

Dear reviewers, thanks for your review. Could you please read the authors’ rebuttal and leave a brief response in the discussion thread? It would help to know which of your main concerns the response addresses and which remain. If your recommendation changes, please briefly explain what led to the change. A few sentences are enough. Thanks for your help!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.