- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents AgentSimArena, a benchmark for evaluating the accuracy and efficiency of existing LLM serving simulators under agent workloads. The benchmark replays fixed agent execution traces while preserving dependencies between consecutive LLM calls, and compares simulator predictions against real vLLM executions under matched deployment configurations.
Update:
After reading the authors' rebuttal, I tend to keep my current score.
Critical strengths
- A standardized comparison of simulators against real serving systems can provide useful guidance for practitioners
- The paper establishes a common evaluation protocol across six existing simulators, aligning workload inputs, deployment constraints, and performance metrics
Critical weaknesses
The paper mainly observes that simulators exhibit different prediction errors across hardware and workload configurations, and that accuracy comes with varying computational costs. I feel this is largely expected given their different performance models, profiling data, and supported mechanisms. The paper primarily quantifies these differences without establishing sufficiently novel or generalizable insights.
Although the paper reports substantial differences in prediction accuracy, it provides limited causal analysis of the underlying reasons.
Despite reporting 240 predictions, the evaluation covers only eight underlying deployment configurations, five arrival rates, and a single real serving engine. The results also lack sufficient repeated measurements and uncertainty estimates.
Questions for authors
Check weaknesses.
3 comments
Response to Reviewer kWS5 (2/2)
W2: Causal analysis of the accuracy differences.
We now trace the errors to their causes (Section 6, Appendix H.6). For all 240 predictions, we split each program's time into time to first token (queueing plus prompt processing) and decode time. For the worst cases, we logged every simulated step.
- APEX on Llama-70B, B200, TP2 (+995% to +1,180%): real vLLM takes 98% of each prompt from its cache. APEX has no prefix caching or chunked prefill. It recomputes the whole prompt (median 13,438 tokens) at every call, in one step that blocks all decoding. Its decode-only steps are accurate (16.4 ms vs 15.8 ms), but prompt steps take 86% of its simulated time.
- LLMServingSim on Llama-70B, B200, two instances (−19% to −24%): the real cache is small (83,312 tokens per instance). It evicts context before the next call can reuse it, so calls queue for minutes. LLMServingSim's prompt processing is too short in all eight deployments (0.36–0.79× real). Here prompt work is large, so shorter prompts shorten the queue: 477 s vs 575 s per program at the highest load.
- Over the 40 settings, APEX and TokenSim are too slow in both phases. Their time to first token is 2.13× and 1.38× real, and their decode time 2.29× and 1.26×. LLMServingSim is at 0.96× in both. Its time to first token includes queueing, so it can look close to real while prompt processing alone is too short.
W3: Deployments, arrival rates, engine and repeated runs.
- Repeats (Section 5.2, Appendix H.2): we repeated all 40 real runs. Mean JCT changed by 2.0% on average and by at most 9.2%. 35 of 40 settings are within 5%. Each simulator's error moves by at most 2 points, and the order is unchanged. In the table below, run 1 is the paper's real runs and run 2 the repeat.
- Deployments: the eight cover two GPU types, three models (dense and mixture-of-experts), tensor parallelism and replication. The most accurate simulator changes between them: DynoSim in four, LLMServingSim in two, TokenSim in two. Adding a deployment needs a configuration file, real runs, and profiles for GPUs a simulator does not yet cover.
- Arrival rates: the five rates go from light load to loads where, in memory-limited deployments, the real cache can no longer hold each program's context (finding 2).
- One engine: all real runs use vLLM 0.19, and Section 7 now states this as a limitation. We leave a comparison against a second serving engine, such as SGLang, to future work.
| Simulator | Mean error, run 1 | Mean error, run 2 |
|---|---|---|
| LLMServingSim 2.0 | 7.1% | 7.0% |
| DynoSim | 10.6% | 9.8% |
| Frontier | 33.8% | 34.8% |
| TokenSim | 39.4% | 39.9% |
| SplitwiseSim | 97.0% | 97.5% |
| APEX | 279.7% | 281.7% |
I appreciate your thorough response. The new error analyses help clarify why the simulators failed, and running the experiments repeatedly on the real system makes readers confident in the overall findings. My remaining concern is whether these findings provide sufficiently novel and generalizable insights beyond diagnosing the evaluated simulators under the tested configurations, which I believe is only partially addressed.
Response to Reviewer kWS5 (1/2)
We thank the reviewer for the review. We traced the errors of all six simulators to three general causes, now in Section 6: missing prompt reuse, memory pressure and step timing. Usual validations miss the first two. A repeat of all 40 real runs leaves the ranking unchanged.
W1: General insights from the error analysis.
We agree that different models give different errors. Our new analysis shows which modeling choices cause them on agent workloads. The pattern holds across all six simulators (table below, 40 settings).
Usual validations miss these findings. Independent requests share at most a common prefix, such as a system prompt, not a program's earlier calls, so they cannot show finding 1. Replay at recorded timestamps hides finding 2: Frontier's mean error drops from 34.4% to 8.2% with its model unchanged. Arena shows both because each call waits for the previous one, in simulation and in real serving. For simulator developers, accurate step timing is necessary but not enough on agent workloads: APEX's decode-only steps match real vLLM (16.4 vs 15.8 ms), yet without prefix caching its mean error is 280%.
ρ is the rank correlation between a simulator's error and real prompt reuse. Near 1, the error grows with reuse. Near 0, it does not depend on it.