- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on the interesting problem of assessing how well AI systems can infer the goals of human users during multi-turn interactions. Towards this goal, the paper introduces a new dataset of human-written, validated goals over existing human-AI conversations, under the key insight that third party observer perception of the human conversant goals is an appropriate target of investigation. They also present experiments with a variety of models, attempting to label (or produce) the human conversant goals labeled within this dataset. These are the two main contributions of the paper.
Critical strengths
The main positive aspect, in my mind, is the initial data collection described in section 3. The procedure is well presented, and can be easily replicated or expanded upon in future studies. The detailed description of the process also leads to confidence in the quality of the data, subject to the limitations described in sections 3 and 4.
Critical weaknesses
The main weakness, in my opinion, has to do with the data collection itself, though I will note a secondary concern about the evaluation process.
The authors themselves identify some of the weaknesses of the proposed method, but it is worth listing that there are significant limitations to the scope of the data collection performed. Among these limitations, I will note the limited scale of the similarity judgments, and the lack of inter-annotator agreement (Fleiss Kappa = 0.33 for the initial set, and Cohen's Kappa = 0.44 for the held-out set). A more general weakness, in my opinion, is the overall somewhat cumbersome process of reconciliation of human judgments in the collection (i.e. the consolidation of human-written goals, described in section 4.2). This process involves developing an LLM judge to generate similarity scores for pairs of goals within a single turn in a single conversation, followed by correlation clustering over the goals, and finally a selection of single goals based on this clustering to clean up the final dataset. My concern is that this is a pretty brittle, difficult to generalize, process; it is difficult to determine how much of this process could be improved, and the details of the experiments are left out; there are no baselines produced for any of the developed components, such as the clustering algorithm or the LLM judge. How much would the rest of the results change, with only minor changes in this process? It is impossible to determine.
My other concern is about the experimental results. It appears that the only baseline used is the human baseline; however, I am wondering whether there are any automated baselines which one could also compare to in addition to this. In particular, because results claim to surpass the human baseline, but this is only due to the filtering process summarized above (and described in section 4.2 of the paper), it is difficult to assess the validity of these results in isolation.
Questions for authors
This is more of a suggestion for future work than a question, but I am wondering whether it would be better to limit the scope of the initial collection, based on these initial results, and achieve a more robust dataset which does not require all the gymnastics in section 4.2 to clean up.
3 comments
Thank you for answering my questions. Please do include some of these results in the final submission of your paper. My score remains as-is but I appreciate the clarifications and believe that they will make the camera-ready submission stronger.
Thank you for your engagement!
Thanks for your engagement! We just wanted to close the loop on the point about the robustness of the procedure: we calculated precision/weighted recall using 2 different random seeds for the randomly-selected cluster representative. The qualitative pattern (e.g. placement of each model on the figure) stayed the same across the two seeds. We will include a figure in the appendix demonstrating this robustness.
Thank you for the positive comments + additional information
We thank the reviewer for their positive comments about the clarity of presentation, replicability of the procedure and high quality of the dataset. Here are our responses to additional comments and questions raised:
Re: inter-annotator agreement: Thanks for raising this point - to be clear, the Cohen’s kappa value between the held-out 4th annotator judgment and the consensus judgment is a baseline, not a ceiling. It suggests that human judgments have some degree of variation, but it does not inherently mean that there is a performance ceiling for an LLM judge. It is possible that a judge can be trained to surpass human accuracy at predicting consensus judgments, e.g. because the judge has access to training examples that the human annotator does not. In fact, we show that our updated judge model achieves a Cohen’s kappa of 0.73 on the human-human test set pairs, outperforming the human baseline of 0.42. See the next section for more details about the judge.
Re: baselines for goal consolidation procedure: We have conducted additional experiments to evaluate and improve the similarity judge (varying base models, loss functions and datasets for finetuning), and report several baselines.
Keeping with our original API judge, we finetuned new judge models to predict the number of annotators who rated a goal pair as expressing the same intention (a value between 0-3), given the history of the conversation up until that point. Again, we evaluated judge performance using Cohen’s kappa between the binarized judge prediction and the majority decision of the annotators. Note that the formula for Cohen’s kappa integrates the likelihood of agreement by chance; a kappa of 0 indicates chance agreement.
Our baselines were as follows:
We experimented with finetuning on 3 different datasets: the train set of 1082 human-labeled examples (human-only), a synthetic dataset that was generated from the API judge models provided with the train set in context (synthetic-only), and both the human and synthetic data. The base models were Qwen 3.8 27 B and Gemma 3 27B. Results are as follows:
Our overall best-performing judge (Qwen finetuned on the human data) outperformed the human baseline and the API judge on the human-human goal pairs, and performed comparably to the human baseline and the API judge on the human-model goal pairs.
We will re-run the analyses with these different components and add a fuller discussion of how they contribute to the outcomes in the revision.
Re: baseline for experimental results: We appreciate the point about baselines for the benchmark itself. Because the task involves producing an open-ended output, as opposed to choosing from a fixed set of options, we are unsure how to operationalize an automated baseline. We are open to suggestions on this front. In any case, in the revision we will modify the language around the human baseline to make it clear that this is a baseline in a relatively weak sense, and should not be taken as evidence that models are “superhuman”.