- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper argues that multi-turn conversations become difficult for LLM agents not only because they are longer, but because they contain more distinct user intents, which the authors call intent multiplicity. To study this, they introduce CloudIntent-Bench, a benchmark of 1,973 decision tasks from real cloud-service support conversations where agents must identify the current intent and choose the appropriate next action. Across several frontier models, performance drops as intent multiplicity increases. Across frontier models, performance drops as intent multiplicity increases and filtering irrelevant past intents also helps, which suggests that interference among multiple intents is a key source of failure in multi-turn agents. I still hold my points (weak reject).
Critical strengths
- The benchmark is built from authentic cloud-service support conversations, making the evaluation setting more realistic than synthetic multi-turn tasks.
- The paper isolates intent multiplicity from dialogue length and provides controlled studies to support its effect on agent performance.
- The evaluation covers multiple frontier models and includes both intent understanding and action selection, providing a useful diagnostic view of multi-turn agent failures.
Critical weaknesses
- Intent multiplicity U is a rather coarse measure of dialogue complexity, which only counts the number of distinct intents and ignores factors such as intent similarity, transition patterns, recency, dependencies. As the authors also acknowledge, conversations with the same U can still have very different levels of difficulty.
- The paper provides limited explanation of why multiple intents cause agent failures. The controlled experiments show that filtering irrelevant history or explicitly providing the current intent improves performance, but this mainly confirms that better context management helps. It does not identify whether the failures come from intent recognition, evidence retrieval, state tracking, or action selection.
- The methodological novelty is limited. The paper mainly introduces a new descriptive metric and benchmark, without proposing a new agent architecture, learning algorithm, or context-management method to address the identified problem.
- The intent annotation used to compute U is not highly reliable. The reported exact intent-count agreement is only 76.4%, while the exact boundary F1 is 0.397, introducing non-negligible noise into the paper’s central explanatory variable.
- The claim that performance decreases as intent multiplicity increases is not consistently supported across the full range of U. The clearest decline mainly appears between U=2 and U=4, while the trend becomes unstable for larger U, partly because the high-intent groups contain very few examples. The results do not support a strictly monotonic relationship.
Questions for authors
- How were the 1,973 decision checkpoints selected from the original support tickets? Please clarify the sampling and filtering criteria, including whether multiple checkpoints can come from the same ticket and whether the selection procedure may bias the distribution of intent multiplicity.
- Why is intent multiplicity computed from annotations produced on the complete ticket rather than only the history visible at the decision checkpoint? Could the authors report how much U changes when the annotator is restricted to the visible history? This seems important because future dialogue may affect the inferred intent structure.
2 comments
W4 Follow-up on the reliability of intent annotations
W4 Follow-up on the reliability of intent annotations
On the 3,196 complete tickets in the held-out set used to validate the intent annotator, the model and human annotations agreed exactly on the number of intents in 76.4% of cases. Of the 753 tickets with different counts, 653 (86.72%) differed by exactly one intent.
To assess how annotation differences affect the paper’s results, we used historical human intent labels to count the distinct intents in the visible history at each decision checkpoint. We then regrouped all 1,973 tasks using these counts and recomputed pass³ with the original agent outputs, scores, and aggregation method.
Action-balanced pass³ (%) after regrouping by human intent counts
| Model configuration | (n=701) | (n=345) | (n=255) |
|---|---|---|---|
| GPT-5.5 medium | 49.84 | 41.32 | 30.09 |
| GPT-5.6 sol medium | 50.91 | 44.49 | 31.07 |
| Qwen3.7-Max | 46.39 | 37.73 | 17.78 |
| DeepSeek-V4-Pro | 47.62 | 37.47 | 12.36 |
| Kimi K3 | 51.23 | 40.20 | 24.44 |
| GLM 5.1 | 47.72 | 35.21 | 16.39 |
| Six-configuration mean | 48.95 | 39.40 | 22.02 |
Despite differences between the model and human annotations, pass³ point estimates declined at both steps from to for all six configurations after regrouping by human intent counts. The paper’s main finding therefore holds under this alternative grouping. Within this range, higher intent counts are associated with lower pass rates.
Clarifications on checkpoint selection, intent annotation, and benchmark scope
We thank the reviewer for recognizing the benchmark’s realism and diagnostic value. We address the questions and weaknesses below.
Q1 Checkpoint selection and sampling bias
We started from over 20,000 real Chinese-language support tickets, removed records with at most three rounds or missing dialogue content, and performed system/automated-message removal, image OCR, and de-identification. We prioritized multi-intent tickets while retaining a proportion of single-intent tickets for comparison. An LLM then generated candidate decision checkpoints and rated their difficulty on a 1–5 scale. We selected candidates scoring at least 3 and excluded low-quality tasks during Gold construction.
The resulting benchmark contains 1,973 tasks from 1,766 tickets, with 1,574/178/13/1 tickets contributing 1/2/3/4 checkpoints, respectively. Checkpoints within the same ticket can involve different current requests, available evidence, and action requirements. Given the relative scarcity of high-U cases, retaining multiple checkpoints helps expand coverage of these scenarios. This targeted construction shapes the U and difficulty distributions, so benchmark composition does not represent natural prevalence in production support traffic.
Q2 Full-ticket annotation and changes in visible-history U
Complete-ticket annotation follows operational practice and matches the input unit used to train the annotation model. Our shared Official Comment explains this choice and reports the comparison with prefix-only annotation.
Across all 1,973 tasks, 67.46% of counts remain unchanged, 451 increase, and 191 decrease. Regrouping preserves the downward trend in six-configuration average performance over U=2–4. The full comparison is provided in the shared comment.
W1 Rationale and scope of U
We use U as an interpretable semantic count to examine how the number of distinct user intents in a dialogue history relates to agent performance, complementing history length. We view U as a perspective for analyzing model performance, rather than a complete measure of dialogue complexity or a predictor of individual-task difficulty. We also explored other factors and composite formulas, but ultimately selected U as the main analysis variable because its simple definition and clear interpretation directly capture the intent multiplicity studied in this work.
W2 Controlled study of intent understanding
The controlled study examines whether helping the model identify the current intent improves its understanding and decisions. Intent Hint preserves the dialogue history and model configuration of Full History, adding only a description of the current request without specifying the action or answer. This yields higher observed Intent Match and action correctness. Focused History instead filters original messages to reduce historical intent interference, with construction and review requirements designed to preserve essential background and evidence while maintaining message wording and order. Together, these interventions provide diagnostic evidence linking intent understanding to agent performance through explicit intent guidance and reduced historical interference.
W3 Benchmark contribution and methodological novelty
Our main contributions are a benchmark for authentic multi-intent support decisions and an empirical analysis of the relationship between intent multiplicity and agent performance. CloudIntent-Bench draws on real support tickets and purposefully covers multi-intent scenarios. In the samples we analyzed from benchmarks using synthetic dialogues, most conversations contained only one annotated intent. Our work expands coverage of multi-intent support scenarios and provides a resource and evaluation perspective for systematically studying agent performance in this setting.
W4 Reliability of intent annotations
We acknowledge that annotation can affect U-group assignments. The prefix-only comparison reported in our shared Official Comment examines the sensitivity of the performance trend to an alternative annotation setup. The persistence of the aggregate decline over U=2–4 provides complementary support for that trend.
W5 Trend scope and high-U nonmonotonicity
At U=1–2, agents have fewer distinct intents to disentangle, which may make identifying the current request easier and limit interference from other intents. As intent multiplicity increases, distinguishing the current request from other needs in the history may become more challenging. At U≥5, the marginal decline in performance may diminish. Meanwhile, the relatively sparse samples in this range make group-level performance estimates more variable. Our main empirical conclusion therefore focuses on the broad downward trend in performance within U=2–4. This interval provides a useful window for examining the relationship between increasing intent multiplicity and agent performance.