acceptodds
Under review as a conference paper at ICLR 2027

CloudIntent-Bench: Intent Multiplicity Challenges LLM Agents in Real-World Cloud Services

Abstract

Large language model (LLM) agents are increasingly deployed in interactive scenarios, such as cloud services, where conversations often involve multiple user requests, challenging the underlying LLMs to reason over evolving dialogue histories. Although existing analyses often attribute lower performance to longer dialogue histories, interaction turn counts alone provide limited insight into which aspects of a conversation drive this degradation. We therefore investigate **intent multiplicity**, defined as the number of distinct user intents identified in a dialogue history by intent annotations. It provides an intermediate characterization between raw conversational context and agent performance, enabling us to analyze agent performance in real-world applications at a more fundamental level. Furthermore, we introduce CloudIntent-Bench, a benchmark of decision tasks derived from authentic cloud service tickets, to systematically examine the LLM-agent performance in different intent multiplicity scenarios. Experimental evaluations show that performance declines as intent multiplicity increases within a transition region, and controlled studies that remove or provide intent information offer complementary evidence for attributing this decline to intent multiplicity. Our work identifies intent multiplicity as a more fundamental perspective for analyzing agent performance and provides a practical benchmark for developing reliable LLM-agent applications in the cloud services domain.

Primary area:
datasets and benchmarks
Keywords:
Intent Multiplicity, LLM-Agent Performance, Cloud Services Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qWzkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that multi-turn conversations become difficult for LLM agents not only because they are longer, but because they contain more distinct user intents, which the authors call intent multiplicity. To study this, they introduce CloudIntent-Bench, a benchmark of 1,973 decision tasks from real cloud-service support conversations where agents must identify the current intent and choose the appropriate next action. Across several frontier models, performance drops as intent multiplicity increases. Across frontier models, performance drops as intent multiplicity increases and filtering irrelevant past intents also helps, which suggests that interference among multiple intents is a key source of failure in multi-turn agents. I still hold my points (weak reject).

Critical strengths
  1. The benchmark is built from authentic cloud-service support conversations, making the evaluation setting more realistic than synthetic multi-turn tasks.
  2. The paper isolates intent multiplicity from dialogue length and provides controlled studies to support its effect on agent performance.
  3. The evaluation covers multiple frontier models and includes both intent understanding and action selection, providing a useful diagnostic view of multi-turn agent failures.
Critical weaknesses
  1. Intent multiplicity U is a rather coarse measure of dialogue complexity, which only counts the number of distinct intents and ignores factors such as intent similarity, transition patterns, recency, dependencies. As the authors also acknowledge, conversations with the same U can still have very different levels of difficulty.
  2. The paper provides limited explanation of why multiple intents cause agent failures. The controlled experiments show that filtering irrelevant history or explicitly providing the current intent improves performance, but this mainly confirms that better context management helps. It does not identify whether the failures come from intent recognition, evidence retrieval, state tracking, or action selection.
  3. The methodological novelty is limited. The paper mainly introduces a new descriptive metric and benchmark, without proposing a new agent architecture, learning algorithm, or context-management method to address the identified problem.
  4. The intent annotation used to compute U is not highly reliable. The reported exact intent-count agreement is only 76.4%, while the exact boundary F1 is 0.397, introducing non-negligible noise into the paper’s central explanatory variable.
  5. The claim that performance decreases as intent multiplicity increases is not consistently supported across the full range of U. The clearest decline mainly appears between U=2 and U=4, while the trend becomes unstable for larger U, partly because the high-intent groups contain very few examples. The results do not support a strictly monotonic relationship.
Questions for authors
  1. How were the 1,973 decision checkpoints selected from the original support tickets? Please clarify the sampling and filtering criteria, including whether multiple checkpoints can come from the same ticket and whether the selection procedure may bias the distribution of intent multiplicity.
  2. Why is intent multiplicity computed from annotations produced on the complete ticket rather than only the history visible at the decision checkpoint? Could the authors report how much U changes when the annotator is restricted to the visible history? This seems important because future dialogue may affect the inferred intent structure.
2 comments
AuthorsOpenReview ↗

Clarifications on checkpoint selection, intent annotation, and benchmark scope

We thank the reviewer for recognizing the benchmark’s realism and diagnostic value. We address the questions and weaknesses below.

Q1 Checkpoint selection and sampling bias

We started from over 20,000 real Chinese-language support tickets, removed records with at most three rounds or missing dialogue content, and performed system/automated-message removal, image OCR, and de-identification. We prioritized multi-intent tickets while retaining a proportion of single-intent tickets for comparison. An LLM then generated candidate decision checkpoints and rated their difficulty on a 1–5 scale. We selected candidates scoring at least 3 and excluded low-quality tasks during Gold construction.

The resulting benchmark contains 1,973 tasks from 1,766 tickets, with 1,574/178/13/1 tickets contributing 1/2/3/4 checkpoints, respectively. Checkpoints within the same ticket can involve different current requests, available evidence, and action requirements. Given the relative scarcity of high-U cases, retaining multiple checkpoints helps expand coverage of these scenarios. This targeted construction shapes the U and difficulty distributions, so benchmark composition does not represent natural prevalence in production support traffic.

Q2 Full-ticket annotation and changes in visible-history U

Complete-ticket annotation follows operational practice and matches the input unit used to train the annotation model. Our shared Official Comment explains this choice and reports the comparison with prefix-only annotation.

Across all 1,973 tasks, 67.46% of counts remain unchanged, 451 increase, and 191 decrease. Regrouping preserves the downward trend in six-configuration average performance over U=2–4. The full comparison is provided in the shared comment.

W1 Rationale and scope of U

We use U as an interpretable semantic count to examine how the number of distinct user intents in a dialogue history relates to agent performance, complementing history length. We view U as a perspective for analyzing model performance, rather than a complete measure of dialogue complexity or a predictor of individual-task difficulty. We also explored other factors and composite formulas, but ultimately selected U as the main analysis variable because its simple definition and clear interpretation directly capture the intent multiplicity studied in this work.

W2 Controlled study of intent understanding

The controlled study examines whether helping the model identify the current intent improves its understanding and decisions. Intent Hint preserves the dialogue history and model configuration of Full History, adding only a description of the current request without specifying the action or answer. This yields higher observed Intent Match and action correctness. Focused History instead filters original messages to reduce historical intent interference, with construction and review requirements designed to preserve essential background and evidence while maintaining message wording and order. Together, these interventions provide diagnostic evidence linking intent understanding to agent performance through explicit intent guidance and reduced historical interference.

W3 Benchmark contribution and methodological novelty

Our main contributions are a benchmark for authentic multi-intent support decisions and an empirical analysis of the relationship between intent multiplicity and agent performance. CloudIntent-Bench draws on real support tickets and purposefully covers multi-intent scenarios. In the samples we analyzed from benchmarks using synthetic dialogues, most conversations contained only one annotated intent. Our work expands coverage of multi-intent support scenarios and provides a resource and evaluation perspective for systematically studying agent performance in this setting.

W4 Reliability of intent annotations

We acknowledge that annotation can affect U-group assignments. The prefix-only comparison reported in our shared Official Comment examines the sensitivity of the performance trend to an alternative annotation setup. The persistence of the aggregate decline over U=2–4 provides complementary support for that trend.

W5 Trend scope and high-U nonmonotonicity

At U=1U=1–22, agents have fewer distinct intents to disentangle, which may make identifying the current request easier and limit interference from other intents. As intent multiplicity increases, distinguishing the current request from other needs in the history may become more challenging. At U≥5U\geq5, the marginal decline in performance may diminish. Meanwhile, the relatively sparse samples in this range make group-level performance estimates more variable. Our main empirical conclusion therefore focuses on the broad downward trend in performance within U=2U=2–44. This interval provides a useful window for examining the relationship between increasing intent multiplicity and agent performance.

AuthorsOpenReview ↗

W4 Follow-up on the reliability of intent annotations

W4 Follow-up on the reliability of intent annotations

On the 3,196 complete tickets in the held-out set used to validate the intent annotator, the model and human annotations agreed exactly on the number of intents in 76.4% of cases. Of the 753 tickets with different counts, 653 (86.72%) differed by exactly one intent.

To assess how annotation differences affect the paper’s results, we used historical human intent labels to count the distinct intents in the visible history at each decision checkpoint. We then regrouped all 1,973 tasks using these counts and recomputed pass³ with the original agent outputs, scores, and aggregation method.

Action-balanced pass³ (%) after regrouping by human intent counts

Model configurationU=2U=2 (n=701)U=3U=3 (n=345)U=4U=4 (n=255)
GPT-5.5 medium49.8441.3230.09
GPT-5.6 sol medium50.9144.4931.07
Qwen3.7-Max46.3937.7317.78
DeepSeek-V4-Pro47.6237.4712.36
Kimi K351.2340.2024.44
GLM 5.147.7235.2116.39
Six-configuration mean48.9539.4022.02

Despite differences between the model and human annotations, pass³ point estimates declined at both steps from U=2U=2 to U=4U=4 for all six configurations after regrouping by human intent counts. The paper’s main finding therefore holds under this alternative grouping. Within this range, higher intent counts are associated with lower pass rates.

Reviewer_dGsTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies LLM agents in the context of multi-turn cloud customer support. It posits that the number of distinct user intents present in the dialogue history is a better predictor of performance than dialogue length. The authors introduce CloudIntent-Bench, which consists of 1,973 checkpoints from real tickets where the agent must determine the current intent, select one of four possible next actions, and compose a response.

Critical strengths
  • CloudIntent-Bench is realistic, based on actual tickets. It uses 1,766 real cloud support tickets rather than simulated users. The references reflect what human staff actually did next.
Critical weaknesses
  • Counts were only validated on full tickets. The annotator also has access to the hidden future messages, which may, in turn, influence how the visible portion is segmented. It needs some sensitivity analysis or fresh human counts on visible histories to establish whether the not passing is an artifact of the measurement
  • The intervention study uses a single model across 100 tasks and gives no confidence interval. It provides no detail on how the distractors were assigned or how the intent hints were constructed.
Questions for authors

What is the precise definition of pass used in the metric?

2 comments
AuthorsOpenReview ↗

Clarifications on pass metrics, prefix-only intent counts, and intervention construction

We thank the reviewer for the careful assessment of the annotation procedure, intervention study, and evaluation metrics. We first answer the question about pass, then address the two weaknesses.

Q1 Definition of pass and aggregation

To clarify what the reported pass metrics measure, we describe the criteria for a single-run pass and how the outcomes from three runs are combined.

A single-run pass requires matching the reference action and satisfying task-specific format, message-reference boundary, and action-dependent missing-information requirements recorded during Gold construction. These are checked by a deterministic verifier.

For three binary outcomes yi1,yi2,yi3y_{i1},y_{i2},y_{i3}, the task-level metrics are

  • Mean pass: (yi1+yi2+yi3)/3(y_{i1}+y_{i2}+y_{i3})/3.
  • pass³: yi1yi2yi3y_{i1}y_{i2}y_{i3}.
  • pass@3: 1−(1−yi1)(1−yi2)(1−yi3)1-(1-y_{i1})(1-y_{i2})(1-y_{i3}).

Here, passi3=1\mathrm{pass}^{3}_i=1 requires success in all three runs and pass@3i=1\mathrm{pass@3}_i=1 requires success in at least one. We first average task-level scores within each Gold-action category, then give the four categories equal weight to account for their uneven representation (Appendix B).

W1 Intent counts on checkpoint-visible histories

Our shared Official Comment reports the analysis using only checkpoint-visible histories for all 1,973 tasks, including the original annotation procedure, the rationale for using complete tickets, count changes, and regrouped performance.

The downward trend over U=2–4 is observed under both full-ticket-projected and prefix-only annotations. Thus, changing the measurement approach does not alter our main conclusion that average agent performance declines as intent multiplicity increases within this range.

W2 Construction of Focused History and Intent Hint

Both interventions used checkpoint-visible dialogue and Gold reference annotations, with GPT-5.6-sol performing generation and model review in separate calls.

Focused History. We construct an ordered subsequence of complete original messages, preserving their wording without summarization. The generator receives the dialogue prefix and Gold intent, constraints, evidence requirements, and action rationale. Candidate distractors are messages supporting resolved or competing historical goals, objects, or workflows that could divert the current decision. Retention rules require the checkpoint message, current-intent messages and necessary background, referent anchors, required evidence, and linked tool-call/return sequences. Additional original messages may be retained to connect the remaining context. Programmatic checks verify message references, chronological order, and mandatory retention. Model review assesses whether the current task and its evidence are preserved and whether removed content meets the distractor criterion.

Intent Hint. We retain Full History and add an active_intent_hint field describing the current request's object, goal, and necessary constraints. The generator receives the dialogue prefix and Gold annotations, including the action and rationale fields. Its instructions restrict the hint to the current request and exclude the reference action, answer, missing-information checklist, and future facts. Format and explicit-leakage checks are followed by model review, revision, and re-review.

Paired results. We further quantified the paired outcomes on the same 100 tasks. Both Focused History and Intent Hint increased action correctness from 55% to 65%, each correcting 17 previously incorrect decisions while introducing 7 errors on previously correct decisions. IntentMatch increased by 0.0744 [0.0371, 0.1163] and 0.2044 [0.1376, 0.2725], respectively. Brackets indicate approximate simultaneous 95% bootstrap confidence intervals, Bonferroni-adjusted across six comparisons.

AuthorsOpenReview ↗

W1 Follow-up on sensitivity to intent measurement

To address the reviewer’s concern that the decline in pass rates may depend on how we measure UU, we ran an additional sensitivity analysis using historical human intent labels assigned on complete tickets. We truncated each label sequence at the decision checkpoint, counted distinct labels in the visible prefix, and regrouped all 1,973 tasks. The two counts agreed on 1,504 of 1,973 tasks (76.23%). Of the 469 tasks with different counts, 402 (85.71%) differed by exactly one intent.

Action-balanced pass³ (%) after regrouping by historical human intent counts

Model configurationU=1U=1 (n=564)U=2U=2 (n=701)U=3U=3 (n=345)U=4U=4 (n=255)
GPT-5.5 medium53.6949.8441.3230.09
GPT-5.6 sol medium56.5650.9144.4931.07
Qwen3.7-Max51.9046.3937.7317.78
DeepSeek-V4-Pro51.7847.6237.4712.36
Kimi K355.8851.2340.2024.44
GLM 5.155.3247.7235.2116.39

Action-balanced pass³ declined at both steps from U=2 to U=4 for all six configurations. The paper’s original finding that higher intent multiplicity is associated with lower pass rates over this range therefore holds under the observed differences between the model-generated and historical human intent counts.

Reviewer_8th4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents CloudIntent-Bench, a benchmark of 1,973 decision tasks derived from authentic, anonymized cloud-service support conversations. It introduces intent multiplicity, the number of distinct user intents in a dialogue history, as a dialogue-level difficulty measure. Experiments across six model configurations show that performance generally declines as U increases, and a small diagnostic study suggests that filtering historical distractors or explicitly supplying the current intent may help.

Critical strengths
  1. The problem is well motivated and practically important. Dialogue difficulty cannot be explained by context length alone: two conversations of the same length may contain very different numbers of user intents, and U is a natural way to capture this.
  2. The benchmark is built from real cloud-service support tickets, making the setting considerably more realistic than synthetic multi-intent conversations.
Critical weaknesses
  1. The data construction is underspecified. The paper gives little detail on ticket filtering, checkpoint selection, preprocessing, languages and translation, or the distribution of service categories. For a benchmark paper this is consequential: without these details, readers cannot judge whether the 1,973 tasks are representative of real support workloads, nor reproduce or extend the benchmark, which undermines its value as a shared resource.
  2. The effect of U is not isolated. (1) U is predicted for all 1,973 tasks by a 32B-parameter model rather than manually verified. Its exact-count agreement is only 76.4% on complete tickets, and its accuracy on the checkpoint-visible histories actually used in the benchmark is unknown. Since a one-count error moves a task into a neighboring U group, substantial label noise may be present in the very variable the paper’s main analysis is built on. (2) The Focused History intervention changes both the intent information and the surrounding context, so the observed gains cannot be attributed to intent reduction alone. A cleaner control would add or remove a single resolved historical intent while keeping the current request, evidence, and context length approximately fixed. The authors acknowledge both issues, but they remain unresolved, and together they weaken the central claim that U, rather than confounded context properties, drives the performance decline.
Questions for authors
  1. The Golden packages are produced through model-assisted drafting and human–LLM review, but their accuracy is not quantitatively validated. How many human reviewers were involved, what was inter-annotator agreement, and how were disagreements resolved?
  2. Some tickets contribute multiple checkpoints. Source-ticket bootstrapping addresses statistical dependence in confidence intervals, but tickets with more checkpoints still receive more weight in task-level point estimates. Could you report one-checkpoint-per-ticket or equal-ticket-weighted results?
  3. Can you report the U-predictor’s accuracy on checkpoint-visible histories (rather than complete tickets), or results on a manually verified subset, to bound the effect of U-label noise on the main trend?
  4. The novelty should be positioned more carefully. Prior work has already studied fragmented and evolving user intents, and CirrusBench also builds on authentic cloud-service tickets. The main contribution here appears to be the dialogue-level measure U and its analysis on real support histories, which is valuable but makes the claim that U offers a “more fundamental perspective” seem overstated.
3 comments
AuthorsOpenReview ↗

Clarifications on Gold quality control, dataset construction, and intent-multiplicity analysis

We thank the reviewer for the careful assessment of measurement validity, benchmark construction, and our contribution. We address the four questions and two weaknesses below.

Q1 Human review and reliability of Gold

We used separate GPT-5.6-sol calls to generate and review Gold references. One human reviewer spot-checked reference quality and whether the reviewing model identified problems and provided reasonable comments. When human and model judgments differed, we incorporated the criteria overlooked by the model into the prompts. Identified invalid tasks were excluded.

This procedure involved single-reviewer spot checks rather than independent annotation by multiple reviewers, so no inter-annotator agreement statistic is available. We therefore report this process as quality control rather than as an accuracy estimate for the final Gold references.

Q2 One-checkpoint-per-ticket sensitivity

Following the reviewer’s suggestion, we retained one checkpoint per source ticket using an outcome-independent fixed hash rule, yielding 1,766 tasks. We recomputed the six-configuration average pass³ using the paper’s four-category action balancing.

Checkpoint selectionU=2U=3U=4U≥5
All checkpoints49.68%37.52%26.45%26.51%
One per ticket49.96%37.52%26.62%25.37%

The U=2-to-U=4 decline is 23.23 percentage points (95% CI [17.70, 28.79]) with all checkpoints and 23.34 points ([17.08, 29.49]) with one per ticket, using 5,000 source-ticket bootstrap resamples. The decreasing pattern also remains in each configuration.

Q3 Accuracy of U in visible histories

Our shared Official Comment reports the visible-history reannotation procedure, count changes, and regrouped performance for all 1,973 tasks. The six-configuration average retains a downward trend over U=2–4.

Q4 Contribution relative to prior work

As discussed in §2, CirrusBench already studies service behavior on authentic cloud tickets. CloudIntent-Bench further examines the intent composition of real support histories and how distinct intent counts relate to agent performance, complementing existing work on service quality and evolving user goals. We position U as a complementary semantic descriptor of agent performance, rather than a predictor of task difficulty.

Some existing benchmarks and datasets use synthetic dialogues generated through interactions between task agents and simulated users. In the τ2\tau^2-Bench and FoodDeliveryService samples we analyzed, 56.5% and 76.8% of dialogues, respectively, were annotated as single-intent. These samples concentrate on lower intent counts and provide limited coverage of higher-intent scenarios. CloudIntent-Bench purposefully covers multi-intent decisions in authentic support histories, providing a resource for systematically studying this setting.

W1 Dataset construction and representativeness

CloudIntent-Bench was constructed from over 20,000 real Chinese-language support tickets. We removed records with at most three dialogue rounds or missing dialogue content, then removed system-generated and automated messages, extracted image text via OCR, and de-identified the dialogues. We prioritized multi-intent tickets while retaining single-intent tickets for comparison. An LLM generated candidate decision checkpoints and rated their difficulty on a 1–5 scale. We selected candidates scoring at least 3 and excluded low-quality tasks during Gold construction. The resulting benchmark contains 1,973 tasks from 1,766 tickets, comprising 1,330 multi-intent tasks and 643 single-intent tasks across 26 product-line or service-routing categories. The largest categories are Elastic Computing (16.52%), SME Applications (16.47%), Cloud Communications (9.33%), Finance (7.81%), Security (6.08%), and Storage (5.32%).

W2 The effect of U is not isolated

U annotation and trend robustness. The shared Official Comment reports that the U=2–4 decline persists under prefix-only reannotation. Separately, repeated annotation produces identical outputs across all evaluated fields on 94.49% of 3,193 jointly valid complete dialogues, supporting procedural stability (§3.3; Appendix A).

Confounding and controlled intervention.

Although Focused History reduces dialogue length, it is designed to retain the information needed to complete the current task. Intent Hint emphasizes the current task using information already present in the visible history, without adding new information. These comparisons therefore provide an indirect diagnostic perspective on the role of intent multiplicity UU.

These are diagnostic results from one model and one output per condition; they do not establish cross-model generalization or a causal effect of UU.

I thank the authors for the detailed response and additional analyses. The clarification of the dataset construction process and the one-checkpoint-per-ticket sensitivity analysis address these concerns. However, the quality control of the Gold references still relies mainly on model generation and review, with only single-reviewer spot checks. The independent or causal effect of (U) remains unclear. I therefore maintain my current score.

Thank you for your reply. Given our staffing constraints, only one human reviewer was available to spot-check the Gold references.

Your question about the independent or causal role of UU raises a narrower question that our data can address. We sought to control for history length by comparing tasks with similar numbers of visible messages. Within each of four overlapping message-count ranges, we grouped tasks by UU and computed action-balanced pass^3 as in Figure 4. Each cell reports the mean across the six model configurations, with the number of tasks in parentheses.

Visible messagesU=2U=2U=3U=3U=4U=4
12–1849.68% (206)31.57% (117)18.99% (43)
17–2340.55% (154)35.53% (107)28.59% (64)
22–2846.02% (70)37.90% (74)27.17% (60)
27–3342.11% (54)37.89% (56)25.99% (44)

In all four message-count ranges, pass^3 declined from U=2U=2 to U=3U=3 and again from U=3U=3 to U=4U=4. At fixed UU, pass^3 showed no consistent trend from shorter to longer message-count ranges. These results support UU as a independent descriptor of performance differences even after accounting for history length, measured by the number of visible messages.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Shared clarification on visible-history intent counts

We thank all three reviewers for raising related questions about how intent multiplicity is measured at decision checkpoints and whether the reported performance trend depends on the annotation context. We address these shared concerns here and refer to this comment in our individual responses.

Original annotation procedure.

We identify semantic intent groups using complete tickets. At each decision checkpoint, U counts the distinct groups represented in the visible history, excluding groups appearing only in later messages. The evaluated agent receives only the checkpoint-visible history. Nevertheless, later messages available to the annotator may influence how earlier messages are interpreted or grouped, motivating the additional analysis below.

Additional annotation procedure.

For all 1,973 tasks, we reannotated the dialogue history up to and including the decision checkpoint, excluding subsequent messages. We used the same annotation model, prompt, and decoding settings as in the original procedure. We then compared the resulting U counts with the original full-ticket-projected counts and regrouped tasks by the new U. Group-level performance was recomputed using the same agent outputs, evaluation scores, and aggregation method.

Changes in intent counts.

We compared the original projected counts with additional annotations based on checkpoint-visible histories for all 1,973 tasks. Counts agree on 1,331 tasks (67.46%), increase on 451, and decrease on 191. Among the 642 tasks with changed counts, 578 (90.03%) differ by only one intent.

Performance after regrouping.

We retained the same agent outputs and frozen evaluation scores and recomputed the U-grouped results using the additional counts. The table reports pass³ averaged across the six evaluated configurations, with equal weighting of the four Gold-action categories within each configuration and U group. Full-ticket projected denotes annotating the complete ticket and then counting only intent groups represented in the checkpoint-visible history. Prefix-only denotes directly annotating the checkpoint-visible history. Each cell reports task count (n) and action-balanced pass³ averaged across six model configurations.

UFull-ticket projected (n, pass³)Prefix-only (n, pass³)
1643, 52.05%467, 52.08%
2630, 49.68%727, 48.54%
3388, 37.52%413, 37.24%
4224, 26.45%271, 32.24%
≥588, 26.51%95, 28.81%

Analysis and conclusion

Despite differences in intent counts and group-level performance between the two annotation approaches, the six-configuration average retains a downward trend over U=2–4. Under prefix-only annotation, average pass³ decreases from 48.54% at U=2 to 37.24% at U=3 and 32.24% at U=4.

Thus, both approaches support the main empirical finding of an association between increasing intent multiplicity and lower average agent performance over U=2–4, although the magnitude of the gap changes.

Why we use complete tickets

We use complete tickets as the annotation unit to examine user intents within the context of the full support process. This follows operational practice, in which staff annotate complete tickets, and matches the complete-ticket inputs used to train the annotation model. The prefix-only analysis provides a complementary check on the sensitivity of the reported trend to this choice.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.