acceptodds
Under review as a conference paper at ICLR 2027

AgentReplay: Token-Wise Trace Replay Is Essential for Fair Serving System Performance Benchmarking

Abstract

LLM-based agents execute multi-turn workflows with interleaved model inference and tool calls, making efficient serving increasingly important. However, evaluating serving optimizations is challenging because identical tasks can produce different execution trajectories. Changes in generated tokens can alter subsequent prompts, tool calls, and reasoning turns, making it difficult to distinguish system improvements from workload variation. Greedy decoding does not guarantee identical outputs, while replaying only sequence lengths loses token information that affects prefix caching and mixture-of-experts (MoE) routing. To address these problems, we propose AgentReplay, a configurable trace record-and-replay framework for agent serving. AgentReplay records input/output tokens, expert selections, request dependencies, and tool durations. During replay, it forces the recorded output tokens while performing normal autoregressive computation, with optional controls for MoE expert selection and tool delays. This allows different systems to execute the same recorded workload while retaining their own batching, scheduling, and parallelization decisions. We further separate trajectory generation from performance evaluation, enabling compatible smaller models to replay long-horizon traces collected with more capable models. Our experiments show that token-wise replay in AgentReplay effectively eliminates workload variation that greedy decoding and length-wise replay cannot avoid, enabling fairer performance comparisons across serving configurations.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Serving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_e9i5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AgentReplay, a trace record-and-replay framework for agent serving. Agent serving interleaves model inferences and tool calls. AgentReplay records input/output tokens, expert selections and request dependencies for the model inference part, as well as tool durations for the tool call part. In addition, it separates trajectory generation from performance generation. This enables smaller models to replay long-horizon traces collected from more capable models. The evaluation compares AgentReplay with greedy decoding and length-wise replay, and shows the benefits of AgentReplay.

Critical strengths
  • AgentReplay is a record-and-replay framework for agent serving. It records both the model inference part and the tool call part. It is useful for evaluating serving optimizations.
  • The evaluation compares AgentReplay with greedy decoding and length-wise replay. The results show AgentReplay can reduce workload variation.
Critical weaknesses
  • The paper does not discuss and evaluate the replay overhead. It would be good to discuss and evaluate the serving performance with and without replay.
Questions for authors
  1. The introduction says that "AgentReplay, a configurable trace record-and-replay framework for vLLM (Kwon et al., 2023)". Is the solution general? Does AgentReplay only work for vLLM? Can the design also be used for other inference engines such as SGLang?

  2. Can you clarify the difference between AgentReplay and XPerf (Wang et al., 2026)? XPerf "records input and output token IDs, invocation dependencies, and tool durations". AgentReplay "records input/output tokens, expert selections and request dependencies and tool durations for the tool call part". Is the difference only whether to record expert selections?

  3. What is the overhead of replay? Does running vLLM with replay introduce any performance overhead compared to running vLLM without replay? Does the overhead affect the evaluation of serving performance optimizations?

Reviewer_kpvPOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AgentReplay addresses a measurement problem in agent serving. Changes in generated tokens can alter later requests, tool calls, and termination, so two executions may perform different amounts of work even when their initial prompts and decoding settings match. The framework records token sequences, invocation dependencies, optional MoE expert selections, and tool durations, then replays that workload under different serving configurations. It retains autoregressive model computation and releases each request when its recorded predecessors finish. Experiments examine variation across repeated coding-agent runs and show how replacing real content with equal-length random tokens changes expert access and execution cost. A further example replays a stronger model's long trajectory on a smaller model. The intended contribution is controlled performance measurement under a fixed workload, with explicit limits when tool contention or changes in generated behavior are part of the system being evaluated.

Critical strengths
  • The problem is well motivated. Different generated tokens can alter subsequent requests, tool calls, and termination, making a faster run an unreliable indicator of a faster serving system. Section 4.1 supports this concern with 50 repeated executions per task. The large variation on SymPy makes the need for workload control concrete.

  • The replay design respects several details that a simpler implementation could miss. Recorded tokens are substituted before updating token-dependent state, and the model still performs its forward pass and native sampling computation. The driver follows invocation dependencies instead of retaining absolute request timestamps. This allows a faster system to release dependent requests earlier while preserving the recorded workflow.

  • The MoE experiments identify useful mechanisms behind benchmark distortion. Section 4.2 connects token content to expert concentration, rank imbalance, and the number of distinct experts accessed. It also distinguishes kernel measurements from full-phase timing. For example, the 1.54 ratio for the slowest prefill rank corresponds to a smaller 1.18 ratio for the full prefill phase.

Critical weaknesses
  • The reduction in time variation is expected when tokens and tool delays are fixed. It does not establish that replay preserves the relative performance of the serving configurations being compared. The paper does not quantify the overhead of token substitution, route restoration, or reading the trace. Such overhead could affect configurations differently, especially at small batch sizes. A matched execution check and an overhead breakdown are important for a tool whose central purpose is trustworthy measurement.

  • The main evaluation uses one target MoE model and two coding tasks. The cross-model experiment uses one completed 132-turn SymPy trajectory. This is enough to demonstrate that the problem exists, but it gives limited evidence about the framework's usefulness across workloads. In particular, prefix caching is a major motivation in Sections 1 and 2, yet the paper does not present a cache-policy comparison showing how token replay changes the estimated benefit of an optimization.

  • Section 4.2 replaces content with random tokens. Some experiments then deliberately place the experts favored by those tokens on one rank or offload those hot experts. These are informative stress cases, and the random-placement result adds useful evidence. However, they do not establish the typical error of replay that retains real prompts and generates meaningful continuations of a fixed length. A comparison with that intermediate control would clarify when full token replay provides a substantial additional benefit.

Questions for authors

Plz see the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.