acceptodds
Withdrawn from ICLR 2027

AtomDialBench: From Aggregate Scores to Atomic Task Profiles for Dialogue Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As organizations increasingly deploy large language models (LLMs) for customer-service review and meeting analysis, model selection demands operation-level evidence rather than merely an aggregate rank. Existing resources offer valuable evaluations but differ in the operations they cover, the outputs they require, and how those outputs are scored. We introduce **AtomDialBench**, a Chinese-English diagnostic benchmark with **16 atomic tasks**, two composite evaluation units, and **3,459** instances. Each task specifies an operation, output requirements, and a scoring procedure; the construction pipeline combines dialogue selection, schema-guided synthesis, and factual checks. Here, atomic means separately specified and scored rather than an independent latent ability. Across **21 LLMs from seven families**, the top five aggregate scores fall within **0.013**, yet task-level rankings differ: the aggregate leader ranks first or joint first on seven of 16 tasks but last on keyword extraction, where it trails the task leader by **0.111**. A 900-response human study yields high inter-annotator agreement (ICC(2,1) ) and a system–human Spearman correlation of . In a controlled comparison with similarly sized training sets, task-aligned fine-tuning yields a higher aggregate score but uneven operation-level changes. Together, AtomDialBench supports aggregate screening, operation-specific comparison, and response-level diagnosis while complementing existing evaluations.

Primary area:
datasets and benchmarks
Keywords:
LLM, Dialogue Understanding, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_urvEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a dialogue understanding evaluation benchmark to assess a model’s ability to understand and analyze a past dialogue. Based on dialogue analysis, the authors propose 16 atomic tasks, including intent understanding, information extraction, attribute extraction, and relationship determination, and also incorporate multi-task combinations and complex instruction constraints. The proposed benchmark comprises 3,459 Chinese and English evaluation instances and evaluates 21 dialogue models. By organizing the dialogue evaluation around the 16 atomic tasks, the paper constructs a fine-grained evaluation system with clear task requirements and detailed scoring criteria; this design enables the identification of specific steps where models make errors. The paper demonstrates the reliability of the scoring and explores the ability of evaluation results to inform model improvements through human consistency checks and targeted fine-tuning. By integrating individual tasks into a detailed evaluation framework, the paper identifies which tasks models perform poorly on; however, it does not fully explain why they perform poorly or why training for one task affects performance on others.

Critical strengths

(1) The tasks proposed in this paper have practical value and can be applied in real-world scenarios such as customer service analysis. The experiments are comprehensive, evaluating multiple models and examining differences between Chinese and English. (2) The paper proposes 16 atomic tasks, shifting the focus of dialogue understanding evaluation from merely reporting overall accuracy to pinpointing the specific steps where models make errors. Additionally, the paper introduces multi-task combinations and complex constraints to compare models’ task execution capabilities from multiple perspectives. (3) The paper combines evaluation with training interventions, using targeted training to explore the proposed evaluation system’s ability to improve model performance.

Critical weaknesses

(1) The scope of task requirements does not fully align with the coverage of actual scoring. Some tasks require the model to output judgments, evidence, rankings, or specific scores, but scoring primarily focuses on the final label; therefore, even if the model provides incorrect evidence, it may still receive a high score. The data generation model and the primary adjudication model use models from the same model family, while manual consistency verification covers only these two models; thus, the current evaluation system may still exhibit systematic scoring biases across model families. (2) Some experiments can demonstrate that changes have occurred but cannot explain the causes of those changes. Single-task and combined-task experiments are not strictly paired based on the same set of dialogues and identical requirements; thus, it is currently impossible to conclude that score differences are due to differences in task combinations. Furthermore, the data volumes and training settings in some training experiments are inconsistent, making it impossible to attribute performance improvements or declines to transfer or interference between tasks.

Questions for authors

(1) How can we account for biases introduced by the data generation model and the scoring model? Since the data generation and primary scoring use models from the same model family, could this give the models being evaluated within that family an advantage? If we were to switch to a scoring model from a different family, would the main conclusions and rankings remain consistent? (2) When models perform worse on combined tasks, is this due to an increase in requirements, or are the questions themselves more difficult? (3) Do changes in performance on other tasks following training for a specific task truly indicate that there is a synergy or interference between these abilities? (4) Since the manual evaluation covered responses from only two models, can it be considered representative of the performance of other models?

Reviewer_44Y4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Authors present AtomDialBench, a benchmark for dialogue analysis broken down into atomic tasks like intent detection and summarization. AtomDialBench also supports bilingual inputs across English and Mandarin. The benchmark defines 16 atomic tasks that are not specifically grounded in the dialogue topic but rather abstract tasks that apply to all dialogues. The authors score multiple LLMs on this benchmark, showing a fairly tight clustering towards the top.

Critical strengths

Aggregating the tasks as suggested in the paper seems useful, as many of the tasks (summarization, info extraction) are separately evaluated across other evaluation benchmarks but are not generally consolidated in a single place.

Critical weaknesses

(1) Novelty is a large issue here. The authors seem to primarily be taking pre-existing benchmarks, filtering them down, and then augmenting them with labels. Most of the labels are evaluated either through hard comparisons or LLM-as-a-judge. Even though it's nice that all these atomic tasks are now in the same place it is rather unconvincing that I cannot just go and instantiate 16 LLM-as-a-judges or simple python scripts and trivially recreate the evaluation suite here.

(2) The judge-to-judge agreement is concerning. Human-to-human agreement seems to be strong (ICC(2,1) of 0.918) but the judge-to-judge agreement is much lower (0.696). If the primary use case of this benchmark is automated judge scoring then the utility may be quite limited based on these statistics.

(3) Critical details seen to be missing from the manuscript, although please correct me and point me to where these details are if I am wrong. Prompts used to evaluate the LLMs, and examples of actual benchmark queries and subsequent LLM outputs/evaluations would be helpful. How abstract categories like "intent" are transformed into concrete queries is unclear; if this is another LLM doing that translation then details about how that LLM is prompted needs to be provided.

Questions for authors

(1) The process by which queries are constructed are unclear. Are queries deterministically determined or generated dynamically by an LLM depending on the actual chunk of dialogue being evaluated? Can prompts for all LLMs used be supplied?

(2) What is the primary practical use of this benchmark and its novelty, given that humans were not even involved in the process other than post-hoc consistency checks? If it is just a number of hard-coded and LLM-judged tasks, then what is stopping someone from trivially reproducing this idea with a few LLM-as-a-judge prompts?

(3) Why were the 16 atomic tasks chosen as is? How are the authors certain that these are a complete representation of the atomic tasks that may be relevant in dialogue analysis?

(4) Can authors explain the gap in human-to-human consistency versus judge-to-judge consistency, and why this should not be a concern for the validity of the benchmark?

(5) Can authors explain what the point of the fine-tuning experiments is, and what the most novel insight gleaned from those experiments might be? The current conclusions seem to be that training on tasks in the benchmark increases performance on a test set, and training on specific atomic tasks also improve performance on those specific tasks with fairly uneven and often negative performance transfer to different tasks. None of these results are very surprising and it is unclear to me what message the authors are trying to send with these experiments.

Reviewer_oPZ5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces AtomDialBench, a Chinese-English diagnostic benchmark designed to evaluate LLMs at the level of individual dialogue-analysis operations rather than relying solely on aggregate scores. The benchmark defines 16 atomic tasks and two composite evaluation units, each with explicit operation definitions, output requirements, and scoring procedures. Using dialogues selected from existing public datasets, the authors construct 3,459 evaluation instances through LLM-based synthesis and automated quality checks.

The authors evaluate 21 LLMs from 7 model families and examine differences between aggregate rankings and task-level performance. They report that models with similar aggregate scores can exhibit different rankings across individual tasks. For validation, three human annotators evaluate 900 responses from two models, yielding an inter-annotator ICC of 0.918 and a system–human Spearman correlation of 0.697. The paper also examines relationships with existing benchmarks and conducts task-aligned fine-tuning experiments, reporting improvements in aggregate performance alongside uneven changes across individual tasks.

The main contribution is a unified, operation-level evaluation framework and an accompanying bilingual benchmark that provides task-specific performance profiles and supports response-level inspection.

Critical strengths
  1. Fine-grained evaluation framework. The paper provides a structured evaluation framework with 16 atomic tasks and explicit output and scoring requirements. Reporting task-level performance alongside aggregate scores allows readers to inspect differences across dialogue-analysis operations and identify specific failure cases.

  2. Broad empirical coverage. The benchmark includes 3,459 Chinese-English evaluation instances and evaluates 21 LLMs from seven model families. The task-level results provide a useful descriptive comparison of model behavior across different dialogue-analysis operations.

Critical weaknesses
  1. Insufficient novelty and lack of justification for the atomic task framework

The central contribution of this paper is not sufficiently distinguished from existing fine-grained dialogue evaluation and task decomposition research. Prior work, including InstructDial and more recent studies such as DEMO, FineDialFact, and VISTA, has already explored decomposing dialogue understanding or evaluation into finer-grained elements, tasks, or factual units. Although these approaches differ in their specific definitions of atomicity, the paper does not adequately explain what new diagnostic capabilities its framework provides beyond existing methods. The absence of a detailed comparison with recent related work makes it difficult to establish the novelty and significance of the proposed benchmark.

More importantly, the selection and organization of the 16 atomic tasks lack sufficient methodological justification. The authors state that the tasks are derived from recurring application requirements, but do not provide a systematic procedure for identifying, selecting, or distinguishing these operations. Several tasks appear conceptually overlapping, particularly within information extraction and condition checking. The reported high correlations among InfoExtract tasks raise further questions about their incremental diagnostic value. Although high correlations do not establish redundancy, the paper provides insufficient evidence that separating these operations yields meaningful information beyond a smaller or alternative task set.

This issue also affects the aggregate scoring scheme. The benchmark assigns equal weights to all 18 evaluation units, meaning that areas divided into more atomic tasks contribute disproportionately to the final score. Without a principled justification for the task taxonomy or weighting scheme, it is unclear whether the resulting aggregate score reflects meaningful differences in dialogue-analysis performance or arbitrary design decisions.

This is a fundamental weakness because the paper's primary contribution depends on the value of its proposed task decomposition, yet neither its necessity nor its advantage over existing evaluation frameworks is convincingly established.

  1. Insufficient evaluation validity and inadequate evidence for the claimed practical benefits

The empirical validation does not adequately establish that AtomDialBench provides reliable task-level measurements or improves model selection.

First, the human evaluation is limited in both scope and transparency. Three computer-science researchers assessed 900 responses from only two models, Qwen3.7-Max and Qwen3.6-Flash. The paper does not provide sufficient information about the annotators' qualifications, language proficiency, or whether they were independent of the authors. More importantly, both evaluated models belong to the Qwen family, while Qwen3.7-Max was used for benchmark instance synthesis and Qwen3.6-Flash served as the main automatic judge. This design raises concerns about the representativeness of the validation sample and potential evaluation bias.

Although human annotators show high agreement, the correlation between automatic and human scores is substantially lower. Agreement is even weaker for the composite evaluation units, with ρ = 0.557 for MTDA and ρ = 0.513 for ICDA. These correlations indicate some positive agreement but do not establish sufficient reliability for fine-grained model comparisons, particularly when the paper emphasizes small differences between models. Furthermore, this evaluation measures agreement on individual responses rather than whether automatic scoring reproduces human judgments of model-level rankings. The study also does not establish the correctness of the LLM-generated reference answers.

Second, the claimed benefit of task-aware model selection is not convincingly demonstrated. In Section 4.3, the authors identify the best-performing model for each task and evaluate its advantage over the aggregate leader using the same benchmark. The reported average difference is only 0.029 across 16 tasks, with seven tasks showing no observed advantage. This procedure does not constitute an independent evaluation of a model-selection strategy, since the selection and comparison are performed on the same data. A held-out evaluation is necessary to determine whether the observed differences translate into actual improvements. Moreover, only one of the 20 adjacent aggregate-ranking comparisons remains statistically significant after Bonferroni correction, further limiting the interpretation of fine-grained rankings.

There are also concerns about scoring robustness. For AT-16, candidate-identity normalization increases the mean F1 from 0.6153 to 0.6906 on the same eligible subset, indicating substantial sensitivity to output representation. The paper does not establish whether its main task-level ranking conclusions remain stable under such reasonable scoring adjustments.

These issues directly undermine the paper's central practical claim: that its atomic task profiles provide sufficiently reliable and actionable evidence for selecting models and diagnosing dialogue-analysis failures. The current experiments demonstrate variations in benchmark scores, but do not convincingly establish that these variations reflect reliable performance differences or lead to better decisions than existing evaluation approaches.

Questions for authors

In Section 3.3, the authors describe automated quality checks for LLM-generated evaluation instances and reference answers. Were any of the reference answers independently verified by human annotators? If so, how many instances were examined, how were they sampled, and what proportion contained factual or annotation errors? I would like to clarify whether the reported human evaluation was entirely separate from the validation of reference-answer correctness.

Regarding dataset construction in Sections 3.2, 3.3, how was independence between training and evaluation data ensured at the source-dialogue level? Since multiple evaluation instances can originate from the same dialogue, were source-dialogue overlaps explicitly checked, particularly for the task-aligned fine-tuning experiments? Additionally, Appendix B describes target quotas for easy and difficult instances based on two reference models. What were the actual difficulty distributions in the final evaluation sets, and how consistently were these quotas satisfied across tasks and languages?

In Section 4.1, 4 models, including 3 of the top-ranked models, were evaluated only on AtomDialBench and excluded from the external benchmark comparisons. What determined this difference in evaluation coverage? How does excluding these models affect the interpretation of the cross-benchmark correlations and comparisons in Section 4.5?

In Section 4.4, the task-aligned fine-tuning results show improvements on most external dialogue benchmarks, but also substantial regressions on PFDial (−0.336) and MultiWOZ slot extraction (−0.168). The business-domain evaluation additionally reports a large decrease on AT-16 (−0.462), potentially associated with an input-field mismatch. Were these regressions investigated through response-level analysis or corrected-format evaluation? It would be useful to know whether they reflect genuine task-performance degradation or inconsistencies in the evaluation setup, as these explanations would lead to different interpretations of the reported transfer results.

I also discovered a hallucination in the references where the author's name is incorrect, so please correct it carefully.

Finally, Figure 4 presents examples of missed information, out-of-scope evidence, and constraint violations as representative failure modes. How were these examples selected? Were these error categories systematically annotated or quantified across models, or were they identified through qualitative inspection of selected responses? This distinction would help clarify whether the benchmark provides systematic failure diagnosis or primarily supports manual inspection of individual errors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.