- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Paper Summary
The paper asks where nondeterminism first appears in interactive agent benchmarks whose evaluation loop also contains LLM components: a user simulator, a judge, and a retrieval environment. It proposes a first-divergence audit. The audit compares replicated transcripts, finds the earliest message that differs given an identical preceding history, and attributes it to a participant and a message kind. Per-turn hazards normalize for which participant speaks first.
Critical strengths
Critical Strengths
- Thorough experimental work. Serving configurations are verified rather than assumed, miss/hit probes isolate the cache mechanism, and arms, losses, and post-hoc analyses are documented in detail.
- Careful claims. The paper is explicit that the audit locates the first observable discrepancy, not its cause, and it makes no claim that leaderboards flip.
- Practical takeaways. Reward stability does not imply interaction stability. Reproducibility should be assessed over the whole evaluation loop. Serving history and cache resets should be reported as part of the evaluation protocol.
Critical weaknesses
Critical Weaknesses
-
Limited novelty. That prefix caching breaks greedy determinism is already documented in the vLLM issue tracker, the SGLang documentation, He & Thinking Machines Lab (2025), and concurrent work (Patodiya, 2026). The remaining contribution is the audit itself, which amounts to locating the first differing message. It is methodologically light for ICLR.
-
The framing overstates the evidence. The title suggests that the evaluator, as an LLM component, is the unstable party. The evidence instead points to serving infrastructure. The user simulator's opening request likely diverges first simply because its prefix is shared across tasks and most often hits the cache. The paper itself acknowledges that this mechanism is not isolated.
-
The practical impact is unclear.
- Reward changes on a minority of tasks.
- The test for an interaction between cache state and model ranking includes zero.
- Pass rates are low (0.10–0.20), models are 7B–32B, and most arms use only 20 tasks × 3 replicates.
- Hosted APIs, the most common evaluation setting, are out of scope.
It remains unclear which evaluation decisions these findings would change.
-
Presentation is hard to follow. The main text depends heavily on arm codes (E4a, E23b, E26, …) and on appendix material. Most analyses are post hoc, so the paper reads more like an experiment log than a focused argument.
Questions for authors
Questions For Authors
- Beyond confirming known cache-induced nondeterminism, what does the audit reveal that a simple cache-on/cache-off comparison would not?
- Can the opening-turn concentration be tied directly to cache hits, for example through per-request hit/miss logs? If so, is it still accurate to say that "the evaluator varies"?
- Is there any setting where these effects change a model comparison or a benchmark conclusion?
- Do the findings extend to larger models, higher pass-rate regimes, or hosted APIs with provider-side caching?
3 comments
Response to Rebuttal
Thank you for your rebuttal. Insteresting work! But from my perspective, it still looks like a experimental log, not a research paper. I will maintain my score.
Thank you for reading our rebuttal and for your feedback. We are glad you found the work interesting.
Response to Reviewer wtcE Review
Thank you for a careful review. We agree that prefix-cache nondeterminism itself is known (we acknowledged that as well in the paper), that serving infrastructure, not the evaluator as a model, causes the model-side divergence, and that the paper reads too much like an experiment log, the revision will replace arm codes with descriptive names, add a glossary and carry the argument in Sections 3–4. Our contribution is a localisation protocol that separates evaluator, agent and environment sources using only transcripts. A new experiment shows why this matters: where a cache on/off comparison shows no change in reproducibility, the audit still finds where divergence starts.
Below we answer your questions.
It is associated with per-request cache state. In a new arm (primary pairing, cache on,
--max-num-seqs 1verified) we logged each response'scached_tokens. Among replicate pairs with identical history, opening turns diverged in 0/22 pairs served the same cached amount and 26/38 served different amounts (all turns: 0/80 vs 53/215). Pairs share runs within tasks, so these are descriptive counts, not independent trials. In the gpt-oss pairing the association is weaker (2/21 vs 10/39).These logs bear on the exposure explanation you suggest, which §4.2 also raised. The opening does not meet differing cache states more often than the agent's first message (63% vs 79% of pairs); it changes its output far more often when it does (26/38 vs 1/27). In these logs, what distinguishes the opening appears to be its response to cache state, not its exposure; we will revise §4.2 accordingly.
Regarding the framing, the revised abstract will state: "The variability originates in serving infrastructure; in our primary pairing it most often first surfaces in the user simulator's output, before the agent acts." It depends on the pairing: with a Hermes-3-8B agent and gemma-3-12b simulator (App. Q), or with gpt-oss-120b, the agent can diverge first.
One quantitative estimate changes. In our variance decomposition on a default deployment, the unexplained greedy-baseline term (0.058) exceeds both role contrasts (0.044 simulator, 0.040 agent); with the cache off it is zero and both contrasts roughly double (0.100, 0.076; §5, App. D). In our primary pairing, default serving thus understated both role contrasts by about half (44% and 53% of their cache-off values). Beyond that, reward is stable on 76–84% of tasks whose transcripts differ (§4.1), so a repeated score does not show the same interaction occurred, and which remedy to apply depends on where divergence starts (point 1). And even greedy single runs of one configuration on airline (50 tasks) scored 0.140–0.200 across five runs (§4.1), so single-run gaps of that size should not be read as model differences without replication. We find no evidence of a change in model ranking (the cache × ranking interaction in App. M includes zero).
The gpt-oss-120b pairing (a larger agent from a new family, with a 32B simulator) has pass rates of 0.350 (cache on) and 0.433 (cache off), a gap within run-to-run noise (21/60 vs 26/60 episodes). The simulator-opening effect appears with the cache on (6/20 tasks) and not with it off (0/20); the cache-off fix does not transfer, since this agent stays non-reproducible. We will present cache disabling as a remedy for the tested configurations only. Hosted APIs remain untested, since provider-side caches cannot always be switched off or inspected. Localisation would apply to their transcripts unchanged; attributing it to the cache would need server-side logs. We see hosted APIs as important future work.
Thank you again for your detailed review and we would be glad to clarify anything further.