- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how co-batching in LLM inference serving affects agents’ tool calls, executed outcomes, and subsequent interaction histories through controlled experiments in AppWorld. It further examines how these behavioral differences impact task success and evaluates the effectiveness of deterministic inference configurations in mitigating them.
Critical strengths
-
Unlike prior work that primarily focuses on differences in model outputs or tool calls, this paper further examines executed tool outcomes, database states, and task success. It provides valuable empirical evidence that co-batching-induced behavioral differences can propagate to actual tool execution and affect task outcomes, contributing to our understanding of agent execution reliability.
-
The paper employs a rigorous experimental design, including history interventions under common database states, reversed execution orders, and same-history controls. These experiments provide convincing evidence that interaction histories can influence subsequent agent behavior.
Critical weaknesses
-
Insufficient isolation of the co-batching effect. The comparison between singleton and co-batch conditions simultaneously changes batch occupancy and companion request composition, making it difficult to disentangle their individual contributions to the observed behavioral differences.
-
Limited evidence for the generality of the findings. The most prominent task-outcome differences (36/97) are observed on states preselected for behavioral forks, which do not represent the prevalence of such effects in general agent workloads. Furthermore, the evaluation on 16 unselected tasks in Section 6 yields inconclusive average effects for both models, limiting the generalizability of the findings.
-
Limited incremental novelty over prior work. Previous studies have already investigated batch-conditioned output variation, agent trajectory divergence caused by inference serving conditions, and the influence of interaction histories on subsequent behavior under common environment states. This paper primarily extends these findings to tool execution and task outcomes. While this provides valuable empirical evidence, it offers limited new mechanistic insights and does not introduce a new mitigation method or general-purpose benchmark.
Questions for authors
-
How can the authors disentangle the effects of batch occupancy and companion request composition on the observed behavioral differences?
-
Given the preselection of fork states and the inconclusive results on unselected tasks, what evidence supports the generalizability of these findings to broader agent workloads?
-
What are the key novel insights of this work beyond prior studies on batch-conditioned output variation, agent trajectory divergence, and history-dependent behavior?
Reply to kxmP
K1. What happens when only the companions change? Changing only the companions produced no observed excess. We kept the focal request fixed and disabled prefix reuse in the AppWorld comparison. At a batch size of four, replacing three identical-input companions with three length-matched unrelated requests produced zero observed excess action disagreement on Qwen and MiniMax. Changing from singleton to four identical inputs gave zero on Qwen and 0.167 on MiniMax on four H100s (family-cluster interval 0 to 0.500, six families). These controls test the particular companion choices in Table R1.
K2. How should we interpret 36/97? We selected states where singleton and co-batched serving had already produced a repeatable action difference that executed cleanly. In 36 of 97 evaluable states, the two branches differed on at least one task check within eight decisions. This rate describes the selected forks. The two branches differed in official success in 17 cases. Both branches had finished in ten of those cases. In the other seven, at least one branch reached the decision limit (§5.1). The ten completed pairs establish observed task consequences. For the separate screen in K3, we used a fixed rule to choose early states before seeing any controlled-batch outcome. That screen required a valid source action and an environment we could reconstruct.
We extended the capped pairs to a longer decision limit on H20 GPUs. This gave seven DeepSeek pairs with complete, repeat-consistent endpoints. Six reached the same final success outcome and one still differed. These are continuations of the same selected pairs, so the original denominator of selected states does not change. This longer-window comparison is new in this response and is reported separately from the capped outcomes.
K3. What evidence extends beyond selected forks? We also ran a separate AppWorld screen whose states were chosen before observing batch outcomes. In 50 of 200 eligible states on each of DeepSeek and MiniMax, the tool return or the measured state after execution differed. On fresh tasks, both average excesses in requirement disagreement are positive, with intervals that include zero. DeepSeek averaged 0.098 (95% CI −0.036 to 0.233, four positive estimates and twelve zeros). MiniMax averaged 0.089 (−0.035 to 0.214, three positive, eleven zero and two missing). A second environment, τ²-bench retail, also showed pre-execution action changes on DeepSeek (22.0%, 95% CI 12.8–31.2%) and MiniMax (34.2%, 23.1–45.4%). Action changes therefore appear in both tool environments and on both main deployments.
K4. What did we learn beyond earlier work? Earlier batch studies measure changes in outputs, labels or correctness. Earlier common-state comparisons, such as compressed versus full contexts at the same AppWorld state (Min et al.), vary the history by design. In our test, both histories come from the agent's own responses to a batch change. On selected states, we restored the same measured database contents and switched which of these histories the agent saw. Its next action changed in 7 of 8 DeepSeek states and all 8 MiniMax states. The same-history controls agreed, including after we reversed the order (§5.3). Restoring the application data alone can therefore miss a consequence of a batch change. Our contribution is this empirical finding, not a new stabilization method.