- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies a confounding factor in benchmarking LLM serving systems for multi-turn agents: identical initial tasks do not necessarily produce identical execution workloads. Differences in generated tokens can change subsequent prompts, tool calls, dependencies, and termination behavior. AgentReplay records input/output tokens, invocation dependencies, optional logical expert selections, and tool durations, then replays the recorded trajectory while retaining normal autoregressive model computation. Its configurable controls enforce output tokens, optionally restore MoE expert IDs, and emulate tool delays, while leaving batching, scheduling, caching, and physical expert placement to the system under evaluation. The paper also demonstrates replaying a trajectory collected with a more capable model on a smaller target model. Experiments show workload variation under greedy decoding, differences in MoE access patterns and execution costs between random-token length-wise replay and token-wise replay, and improved timing repeatability under fixed-trace replay.
Critical strengths
-
The paper addresses a meaningful benchmarking problem and separates system efficiency from changes in agent behavior. Section 4.1 shows substantial variation in token counts, turns, and tool calls despite greedy decoding. AgentReplay fixes the recorded trajectory without replacing model execution with stored responses. It also appropriately preserves scheduling, cache-management, and physical-placement decisions as outcomes of the system being evaluated. The authors explicitly distinguish improved repeatability from speedup, avoiding the misleading interpretation that executing a different workload more quickly demonstrates a system improvement.
-
The MoE experiments provide informative mechanistic evidence. Section 4.2 connects token content to expert concentration and then examines its interaction with expert parallelism, tensor parallelism, and offloading. The observed bias can have different directions: at batch size 32, length-wise replay yields a full EP prefill-phase time of 1.18 times the token-wise reference, but a TP decode MoE-kernel time of 0.84 times the reference. These results substantiate the narrower, important claim that equal sequence lengths do not imply equivalent MoE execution workloads.
Critical weaknesses
-
Replay fidelity and measurement overhead are not sufficiently validated for a benchmarking framework. Section 3.2 describes a sensible token-replacement point, but the paper does not show implementation-level tests establishing that replay maintains the intended token state and prefix/cache keys across batch reordering and turn boundaries. This does not require different cache policies to achieve identical hit rates; rather, it requires the supplied token history and legal reuse opportunities to match the trace definition. Router replay is presented as a separate contribution, yet Figures 4–6 do not clearly specify whether it is enabled or report recorded-versus-enforced expert-ID agreement and an on/off comparison across configurations. The handling of gate weights after expert IDs are forced is also insufficiently specified. Finally, the overhead of trace lookup and token/router injection is not reported. These omissions do not demonstrate an incorrect implementation, but they limit confidence that the framework controls the intended workload without materially perturbing the measured execution.
-
The experiments establish useful counterexamples more convincingly than they establish reliable practical serving-system comparisons. The length-wise baseline in Section 4.2 replaces content with random tokens. This proves that length matching alone is insufficient in that setting, but it does not quantify the bias of a more realistic baseline that retains recorded inputs and allows natural generation while enforcing output lengths. Some placement and offloading configurations are also deliberately constructed around the random-token hotspots; these are useful diagnostic cases, but their effect sizes should not be generalized without further evidence. In Figure 3(c), output tokens and tool durations are fixed together, so the reduction in cumulative-time CV cannot be attributed to token replay alone. Fixed tool delays can reduce CV both by removing tool-time variability and by adding a constant to its denominator, as the authors acknowledge. Although Figures 5–6 include full-phase measurements, the paper does not demonstrate how replay changes a paired comparison between actual serving configurations under the same agent trace and arrival policy. Missing hardware/software specifications and repeat counts for Figures 4–6 further limit reproducibility.
Questions for authors
1Can you provide replay-consistency tests covering batch reordering, turn boundaries, and prefix/cache-key updates, together with token/router injection overhead measurements, especially for small-batch decoding? The objective is to verify the intended reuse opportunities, not to require identical cache-hit outcomes across different policies.
2.Was router replay enabled in Figures 4–6? How are gate weights handled after expert IDs are forced, and how are routing events aligned across prefill and decode? Please report recorded-versus-enforced ID agreement and one on/off comparison using the existing TP/EP configurations.
3.With the recorded graph and inputs held fixed, what are the model-only and model-plus-tool timing means, standard deviations, and CVs when output-token replay and tool-duration replay are enabled separately and together? This would distinguish their contributions to repeatability.
4.Can you supplement the existing trace with a length-wise baseline that retains real inputs and enforces only output length, and report one paired phase-level or end-to-end cost comparison between actual serving configurations? Please also specify the hardware, software versions, and measurement repeat counts.