- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a dialogue understanding evaluation benchmark to assess a model’s ability to understand and analyze a past dialogue. Based on dialogue analysis, the authors propose 16 atomic tasks, including intent understanding, information extraction, attribute extraction, and relationship determination, and also incorporate multi-task combinations and complex instruction constraints. The proposed benchmark comprises 3,459 Chinese and English evaluation instances and evaluates 21 dialogue models. By organizing the dialogue evaluation around the 16 atomic tasks, the paper constructs a fine-grained evaluation system with clear task requirements and detailed scoring criteria; this design enables the identification of specific steps where models make errors. The paper demonstrates the reliability of the scoring and explores the ability of evaluation results to inform model improvements through human consistency checks and targeted fine-tuning. By integrating individual tasks into a detailed evaluation framework, the paper identifies which tasks models perform poorly on; however, it does not fully explain why they perform poorly or why training for one task affects performance on others.
Critical strengths
(1) The tasks proposed in this paper have practical value and can be applied in real-world scenarios such as customer service analysis. The experiments are comprehensive, evaluating multiple models and examining differences between Chinese and English. (2) The paper proposes 16 atomic tasks, shifting the focus of dialogue understanding evaluation from merely reporting overall accuracy to pinpointing the specific steps where models make errors. Additionally, the paper introduces multi-task combinations and complex constraints to compare models’ task execution capabilities from multiple perspectives. (3) The paper combines evaluation with training interventions, using targeted training to explore the proposed evaluation system’s ability to improve model performance.
Critical weaknesses
(1) The scope of task requirements does not fully align with the coverage of actual scoring. Some tasks require the model to output judgments, evidence, rankings, or specific scores, but scoring primarily focuses on the final label; therefore, even if the model provides incorrect evidence, it may still receive a high score. The data generation model and the primary adjudication model use models from the same model family, while manual consistency verification covers only these two models; thus, the current evaluation system may still exhibit systematic scoring biases across model families. (2) Some experiments can demonstrate that changes have occurred but cannot explain the causes of those changes. Single-task and combined-task experiments are not strictly paired based on the same set of dialogues and identical requirements; thus, it is currently impossible to conclude that score differences are due to differences in task combinations. Furthermore, the data volumes and training settings in some training experiments are inconsistent, making it impossible to attribute performance improvements or declines to transfer or interference between tasks.
Questions for authors
(1) How can we account for biases introduced by the data generation model and the scoring model? Since the data generation and primary scoring use models from the same model family, could this give the models being evaluated within that family an advantage? If we were to switch to a scoring model from a different family, would the main conclusions and rankings remain consistent? (2) When models perform worse on combined tasks, is this due to an increase in requirements, or are the questions themselves more difficult? (3) Do changes in performance on other tasks following training for a specific task truly indicate that there is a synergy or interference between these abilities? (4) Since the manual evaluation covered responses from only two models, can it be considered representative of the performance of other models?