acceptodds
Under review as a conference paper at ICLR 2027

The Evaluator Also Varies: Localizing Nondeterminism in Interactive Agent Benchmarks

Abstract

Interactive agent benchmarks can use large language models both as agents and as components of the evaluation system. When repeated evaluations disagree, the variation may originate in the simulated user or environment before it appears in the agent’s behavior. We introduce a first-divergence audit that identifies the earliest disagreement between replicated interactions with identical preceding recorded histories, locating it in the agent, simulated user, or environment. Across 100 airline and retail tasks on τ²-bench, five greedy vLLM replicates per task reveal transcript divergence on all tasks, but reward variation on only 20. The first discrepancy appears in the simulated user on 83 of the 100 tasks. In banking, disagreements persist in environment outputs after prefix caching is disabled on the agent and simulator. Comparisons of serving configurations reveal different patterns of residual divergence. A complementary ALFWorld experiment shows that preceding serving history can change trajectories without changing aggregate success, while cache resets restore agreement in the tested conditions. Together, these results show that stable benchmark rewards can conceal an unstable evaluation process. Reproducibility must therefore be assessed across the complete interaction, including the simulated user, environment, and serving protocol.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Interactive Agent Benchmarks, Evaluation Reproducibility, Inference nondeterminism, user simulation, prefix caching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wtcEOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

Paper Summary

The paper asks where nondeterminism first appears in interactive agent benchmarks whose evaluation loop also contains LLM components: a user simulator, a judge, and a retrieval environment. It proposes a first-divergence audit. The audit compares replicated transcripts, finds the earliest message that differs given an identical preceding history, and attributes it to a participant and a message kind. Per-turn hazards normalize for which participant speaks first.

Critical strengths

Critical Strengths

  • Thorough experimental work. Serving configurations are verified rather than assumed, miss/hit probes isolate the cache mechanism, and arms, losses, and post-hoc analyses are documented in detail.
  • Careful claims. The paper is explicit that the audit locates the first observable discrepancy, not its cause, and it makes no claim that leaderboards flip.
  • Practical takeaways. Reward stability does not imply interaction stability. Reproducibility should be assessed over the whole evaluation loop. Serving history and cache resets should be reported as part of the evaluation protocol.
Critical weaknesses

Critical Weaknesses

  1. Limited novelty. That prefix caching breaks greedy determinism is already documented in the vLLM issue tracker, the SGLang documentation, He & Thinking Machines Lab (2025), and concurrent work (Patodiya, 2026). The remaining contribution is the audit itself, which amounts to locating the first differing message. It is methodologically light for ICLR.

  2. The framing overstates the evidence. The title suggests that the evaluator, as an LLM component, is the unstable party. The evidence instead points to serving infrastructure. The user simulator's opening request likely diverges first simply because its prefix is shared across tasks and most often hits the cache. The paper itself acknowledges that this mechanism is not isolated.

  3. The practical impact is unclear.

    • Reward changes on a minority of tasks.
    • The test for an interaction between cache state and model ranking includes zero.
    • Pass rates are low (0.10–0.20), models are 7B–32B, and most arms use only 20 tasks × 3 replicates.
    • Hosted APIs, the most common evaluation setting, are out of scope.

    It remains unclear which evaluation decisions these findings would change.

  4. Presentation is hard to follow. The main text depends heavily on arm codes (E4a, E23b, E26, …) and on appendix material. Most analyses are post hoc, so the paper reads more like an experiment log than a focused argument.

Questions for authors

Questions For Authors

  1. Beyond confirming known cache-induced nondeterminism, what does the audit reveal that a simple cache-on/cache-off comparison would not?
  2. Can the opening-turn concentration be tied directly to cache hits, for example through per-request hit/miss logs? If so, is it still accurate to say that "the evaluator varies"?
  3. Is there any setting where these effects change a model comparison or a benchmark conclusion?
  4. Do the findings extend to larger models, higher pass-rate regimes, or hosted APIs with provider-side caching?
3 comments
AuthorsOpenReview ↗

Response to Reviewer wtcE Review

Thank you for a careful review. We agree that prefix-cache nondeterminism itself is known (we acknowledged that as well in the paper), that serving infrastructure, not the evaluator as a model, causes the model-side divergence, and that the paper reads too much like an experiment log, the revision will replace arm codes with descriptive names, add a glossary and carry the argument in Sections 3–4. Our contribution is a localisation protocol that separates evaluator, agent and environment sources using only transcripts. A new experiment shows why this matters: where a cache on/off comparison shows no change in reproducibility, the audit still finds where divergence starts.

Below we answer your questions.

  1. What does the audit reveal that a cache-on/cache-off comparison would not?
  • Separating evaluator from agent (new). With gpt-oss-120b as agent and Qwen3-32B as simulator (20 airline tasks × 3 greedy replicates; one pairing), 0/20 tasks reproduce with the cache on and 0/20 with it off. The audit shows that with the cache off the simulator's opening is identical in all 20 tasks and every remaining first divergence is at the agent (17 at its first message). Consistent with this, six byte-identical greedy requests to the cache-off gpt-oss server returned six distinct outputs.
  • A residual no serving flag reaches. In banking, 12 tasks still diverge with the cache off; the audit places all 12 at an environment tool result, traced to tied BM25 scores.
  • With vLLM's batch-invariant mode, 9/20 tasks reproduce, and all 11 remaining first diverge at the simulator, none at the agent (Table 1); checking agent determinism alone would pass.
  1. Can the opening-turn concentration be tied to cache hits? Is "the evaluator varies" still accurate?

It is associated with per-request cache state. In a new arm (primary pairing, cache on, --max-num-seqs 1 verified) we logged each response's cached_tokens. Among replicate pairs with identical history, opening turns diverged in 0/22 pairs served the same cached amount and 26/38 served different amounts (all turns: 0/80 vs 53/215). Pairs share runs within tasks, so these are descriptive counts, not independent trials. In the gpt-oss pairing the association is weaker (2/21 vs 10/39).

These logs bear on the exposure explanation you suggest, which §4.2 also raised. The opening does not meet differing cache states more often than the agent's first message (63% vs 79% of pairs); it changes its output far more often when it does (26/38 vs 1/27). In these logs, what distinguishes the opening appears to be its response to cache state, not its exposure; we will revise §4.2 accordingly.

Regarding the framing, the revised abstract will state: "The variability originates in serving infrastructure; in our primary pairing it most often first surfaces in the user simulator's output, before the agent acts." It depends on the pairing: with a Hermes-3-8B agent and gemma-3-12b simulator (App. Q), or with gpt-oss-120b, the agent can diverge first.

  1. Does this change a model comparison or benchmark conclusion?

One quantitative estimate changes. In our variance decomposition on a default deployment, the unexplained greedy-baseline term (0.058) exceeds both role contrasts (0.044 simulator, 0.040 agent); with the cache off it is zero and both contrasts roughly double (0.100, 0.076; §5, App. D). In our primary pairing, default serving thus understated both role contrasts by about half (44% and 53% of their cache-off values). Beyond that, reward is stable on 76–84% of tasks whose transcripts differ (§4.1), so a repeated score does not show the same interaction occurred, and which remedy to apply depends on where divergence starts (point 1). And even greedy single runs of one configuration on airline (50 tasks) scored 0.140–0.200 across five runs (§4.1), so single-run gaps of that size should not be read as model differences without replication. We find no evidence of a change in model ranking (the cache × ranking interaction in App. M includes zero).

  1. Larger models, higher pass rates, hosted APIs?

The gpt-oss-120b pairing (a larger agent from a new family, with a 32B simulator) has pass rates of 0.350 (cache on) and 0.433 (cache off), a gap within run-to-run noise (21/60 vs 26/60 episodes). The simulator-opening effect appears with the cache on (6/20 tasks) and not with it off (0/20); the cache-off fix does not transfer, since this agent stays non-reproducible. We will present cache disabling as a remedy for the tested configurations only. Hosted APIs remain untested, since provider-side caches cannot always be switched off or inspected. Localisation would apply to their transcripts unchanged; attributing it to the cache would need server-side logs. We see hosted APIs as important future work.

Thank you again for your detailed review and we would be glad to clarify anything further.

Reviewer_wtcEOpenReview ↗

Response to Rebuttal

Thank you for your rebuttal. Insteresting work! But from my perspective, it still looks like a experimental log, not a research paper. I will maintain my score.

Thank you for reading our rebuttal and for your feedback. We are glad you found the work interesting.

Reviewer_SAEoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates where variation first arises when an LLM agent is evaluated against an LLM user simulator (and, in some domains, an LLM judge or a retrieval environment) with sampling disabled. The authors propose a first-divergence audit: replicate transcripts are compared only while their recorded histories agree, the first differing message is attributed to the agent, simulated user or environment, and a per-turn hazard normalizes for the number of turns at risk. On τ²-bench with Qwen3-30B-A3B as agent and Mistral-7B as simulator, all tasks yield non-identical transcripts across five greedy replicates, yet reward changes on only 8–12 of 50 tasks per domain. For 83 of 100 tasks the first divergence occurs in the simulated user's opening utterance. Disabling prefix caching removes nearly all divergence, except in the banking domain, where unpinned tie-ordering in BM25 retrieval produces a residual. The authors compare several serving controls (batch-invariant kernels, request serialization, SGLang deterministic mode, cache disabling) and show on ALFWorld that preceding serving history alters trajectories without altering pass rate, while a cache reset restores agreement in the tested conditions.

Critical strengths

For this paper, the mechanistic evidence is rigorous. The study includes a 2×2 design over two independent engines and cache settings, a single-model ALFWorld transfer experiment, a request-level miss/hit probe, and a serving-history experiment with an exact control . The authors also explicit mention about which configurations were asserted rather than verified.

Critical weaknesses

The evidence base is small and unevenly verified, and the exposition is difficult to follow. Most contrasts use 20 tasks and three replicates, with small open-weight models (7B–32B) and a weak 7B simulator that is not the current benchmark practice. Regarding to hosted APIs, where most agent evaluations are conducted, is out of scope.

Questions for authors
  1. E4a (serialization only asserted) has simulator hazard 0.243 and E26 (verified) has 0.003. Is serialization the cause of the difference, and how much should the earlier arms be trusted?
  2. Can you log cache hits and misses per request for at least one arm, to show the opening turn is the one that meets partly-cached state?
1 comment
AuthorsOpenReview ↗

Response to Reviewer SAEo Review.

Thank you for your careful review and positive assessment of the mechanistic evidence. We agree that the presentation needs improvement. We will replace arm codes with descriptive names, add a glossary, and bring the key definitions and comparisons into Sections 3–4 and this is one of the common point from all reviewers and we will surely work on this.

Your carefully understood your questions and performed new experiments and here is the more information about the same.

  1. Serialization and reliability of the earlier arms : The new evidence shows that verified serialization alone is not sufficient to reproduce E26's near-complete agreement. We ran 20 airline tasks with three greedy replicates using E26's documented settings: vLLM 0.11.0, prefix caching enabled, one serving replica, the same agent–simulator pairing, and --max-num-seqs 1 verified from the launch log. Only 2/20 tasks reproduced; 14 first diverged at the simulator and four at the agent, with 13 at the simulator's opening.

This does not establish why E26 reproduced 48/50 tasks. E26 used 50 tasks and five replicates, and the runs may differ in preceding traffic and unrecorded serving state. Without per-request cache records for E26, we cannot determine the explanation. We will report it as an observation whose high agreement was not reproduced, not as evidence that serialization resolves divergence.

The earlier arms remain evidence of the transcript and reward outcomes observed under their recorded settings, but cannot establish a causal effect of serialization where it was only asserted. The primary 100-task results, the pooled opening-turn audit, and the original cache-off comparisons do not depend on E26 (Appendices A, D and E). We will clearly label each arm's verification status and revise the interpretation of the serialization comparison. Thank you once again for this question.

  1. Per-request cache measurements: Yes. In the same new arm, we enabled --enable-prompt-tokens-details and recorded each response's cached_tokens. We compared replicate pairs only while their preceding recorded histories agreed.
\hline \text{Turn} & \text{Same cached-token count} & \text{Different cached-token count} \\\\ \hline \text{Simulator opening} & 0/22 & 26/38 \\\\ \text{Simulator later turns} & 0/14 & 5/39 \\\\ \text{Agent first message} & 0/7 & 1/27 \\\\ \text{Agent later messages} & 0/37 & 21/111 \\\\ \hline \text{All turns} & 0/80 & 53/215 \\\\ \hline \end{array}$$ Entries are divergent pair-turn comparisons / at-risk pair-turn comparisons. Both requests in all 60 opening-turn pairs were partial cache hits. All 53 observed pairwise divergences occurred when cached-token counts differed; none occurred among the 80 comparisons with matching counts. These comparisons reuse runs within tasks, so we report the counts descriptively rather than treating them as independent trials. Different cached-token counts occurred in 38/60 (63%) opening pairs and 27/34 (79%) agent-first-message pairs. Among these, outputs differed in 26/38 and 1/27, respectively. So the opening-turn concentration is not explained by differing cached amounts being more frequent; this does not isolate model, prompt or position effects. These measurements show partial cache reuse at the opening and an association with divergence; they do not establish a causal effect of the cached/fresh boundary. Matching counts do not establish identical full cache state. In the new gpt-oss pairing, opening divergences occurred with both matching and differing counts (2/21 and 10/39), so matching counts do not guarantee agreement. 3. Evidence size, larger models and hosted APIs: The primary experiments cover 50 airline and 50 retail tasks with five replicates each; the pooled audit includes 408 task–arm observations across 18 configurations. To address model scale and simulator choice, we added gpt-oss-120b with Qwen3-32B as simulator, using 20 airline tasks and three greedy replicates per cache condition. The first divergence was at the simulator's opening in 6/20 tasks with caching enabled and 0/20 with it disabled. However, all 20 cache-off tasks first diverged at the agent; a separate cache-off probe returned six distinct outputs from six identical requests. Pass rates were 0.350 and 0.433, respectively. This extends the evidence beyond the 7B simulator while showing that cache disabling is not a general remedy. We will restrict the cache-off reproducibility finding to the original tested configurations, with banking's environment-path exception. One additional pairing does not establish broad model-family coverage. Hosted APIs remain untested. The transcript-based audit could be applied where replicated interaction records are available, but our serving-control findings should not be assumed to transfer to provider-managed systems.We see this as a natural direction for future work Thank you again and please let us know if you have any further questions.
Reviewer_vru8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper aims to locate the sources of uncertainty of interactive agents from agent, simulated user and environment. It proposes a first-divergency audit to locate whereh the repeated runs first diverse and cliams that the reproducibility need to be assessed across the entire evaluation loop. The evalaution cost is low and could find the unstable cases. However, it can only see the first diverse and easily affected by sequence. The experiments scale is reletively low.

Critical strengths
  • The research direction is interesting instead of focusing on the final score, this paper focus on evaluating the instability during interactive agent workflow.
  • The evaluation cost it's cheap. Only using the existing transcripts can locate the first diverse and acquire the score.
Critical weaknesses
  • The paper uses too many experiment codes and numbers. Many key details are in long appendices, so the main argument is hard to follow.
  • The component that speaks first is more likely to be blamed, and later turns are excluded. So this method cannot show which component is less stable.
  • Model coverage is limited. The paper tests only four model pairings, all using 7B–32B open models with limited model families and configurations.
Questions for authors
  • How can the audit avoid first-speaker and censoring bias?
  • How do the results change across model types and families?
1 comment
AuthorsOpenReview ↗

Response to Reviewer vru8 Review.

Thank you for your review, and for recognising both the direction (looking inside the interaction rather than only at the final score) and the low cost of the audit. We are also sorry that paper was hard to follow, In the revised version we will keep all the information but replace arm codes with descriptive names, add a short glossary and make sure that paper is easy to follow.

Thank you also for your questions and criticisms, which were genuinely useful to us. Here is an attempt to answers your questions:

1: The audit does not simply count who diverges first; it normalises divergence events by turns at risk, conditional on identical preceding history. In Appendix A the simulator's opening hazard is 0.556 against 0.066 for the agent's own first message, so the concentration is specific to the simulator's opening. This is descriptive: the agent's first turn is conditional on surviving the simulator's opening, and the endpoints differ in model and prompt. In ALFWorld, where the agent is the only model, none of the 20 games diverges at its first action (Appendix L), and in our new 120B experiment below all 20 cache-off tasks first diverge at the agent, which generates second. So divergence is not simply attributed to whoever speaks first, though position effects are not fully isolated.

We agree on censoring. We reanalysed the existing transcripts on every pair of replicates separately, so two runs that agree at the opening stay eligible for later turns even when another replicate has diverged. Later-turn evidence grows from 334 to 1,757 simulator turns and from 756 to 4,109 agent turns after the agent's first message (task-bootstrap 95% CIs):

\hline \text{Turn} & \text{Divergence rate} & \text{95\% CI} \\\\ \hline \text{Simulator opening} & 0.449 & [0.409,\ 0.487] \\\\ \text{Agent's own first message} & 0.046 & [0.025,\ 0.071] \\\\ \text{Simulator later turns} & 0.111 & [0.085,\ 0.142] \\\\ \text{Agent after its first message} & 0.151 & [0.128,\ 0.175] \\\\ \hline \end{array}$$ The rates differ because the original audit compares all replicates jointly, while the new analysis compares each pair separately. This lets us examine more later turns, but each pair still stops at its first divergence because the subsequent histories are no longer identical. We agree the audit cannot rank components by intrinsic stability, on later turns the agent's hazard is in fact slightly higher. We will state this in the introduction and limitations. Our contribution is narrower: the audit identifies **where reproducibility is first observably lost**, not its cause. 2. The paper evaluates four pairings: three agent configurations from two families (Qwen3 MoE, Qwen3 dense, Llama-based Hermes-3) and three simulator families (Mistral, Qwen, Gemma; Table 13). Following your suggestion, we added **gpt-oss-120b as the agent with a Qwen3-32B simulator**, a third agent family and a much larger agent (20 airline tasks, three greedy replicates, cache on vs off): $$\begin{array}{lcc} \hline \text{Result} & \text{Cache on} & \text{Cache off} \\\\ \hline \text{Pass rate} & 0.350 & 0.433 \\\\ \text{First divergence at simulator opening} & 6/20 & 0/20 \\\\ \text{First divergence at agent} & 14/20 & 20/20 \\\\ \hline \end{array}$$ With caching on, six tasks first diverge at the simulator's opening; with it off, none do and all 20 first diverge at the agent. Unlike our original configurations, this agent endpoint stays non-reproducible with the cache off: six identical requests gave six distinct outputs. In our original experiments disabling the cache restored reproducibility (except banking's retrieval residual), so we will present it as a remedy for the tested configurations only. **Reproducibility is 0/20 in both conditions** while the first discrepancy moves, which is what the audit adds beyond a single reproducibility rate. The pass rate (0.350–0.433) also shows the simulator-opening effect is not confined to low-pass-rate settings. This is one additional pairing, not a comprehensive family study. The primary experiments cover **50 airline and 50 retail tasks with five replicates each**; the pooled audit includes **408 task–arm observations across 18 configurations**; and **7,065 conversations were dispatched** overall, including pilots and failed runs (Appendix V). The 20-task comparisons are matched on task IDs; for example, the primary airline comparison moves from **0/20 to 20/20** reproducible with the cache off (exact McNemar **p = 1.9 × 10⁻⁶**; Table 6). We hope that we were able to answer your queries, please let us know if you have any further questions we would be very happy to clarify. Thank You for your review once again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.