- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces a multimodal dataset for studying social intentions in natural interactions, along with three tasks defined on it: apparent intention estimation, social involvement identification, and topic changes.
The data were recorded during a real workshop at an international conference with 32 participants, and it comprises two 30-minute mingling sessions in which participants moved and talked freely, without assigned partners or topics. Participants completed personality and demographic surveys beforehand, and reported their goals and mood before each session and whether they achieved them afterwards. Annotators, who also completed personality questionnaires, watched 30-second clips and described what a given participant seemed to be trying to achieve (the description) and why (the explanation). The aim is to keep multiple interpretations per moment and relate them to each annotator's profile.
Critical strengths
The data come from a real networking event and include several synchronized modalities (video, individual audio, wearable sensors). Participants reported their own goals before and after each session, and a follow-up survey three months later recorded which connections they maintained. Annotators, who also filled in personality questionnaires, described what they believed a participant's intention was and why.
To the best of my knowledge, no existing dataset combines all of this. Such data is expensive and difficult to collect. This is the main reason I see the paper as worth considering despite its weaknesses.
Critical weaknesses
-
The paper is hard to follow: the reader has to go back and forth between the main text and the supplementary material to understand the tasks and results.
-
The introduction presents many theoretical concepts, but finally it does not present a concrete task definition. The central task, Apparent Intent Inference, is not clearly specified in the main paper: what is the input, what is the output, and what are the metrics?
-
The proposed PAS metric does not distinguish clip-grounded narratives from generic ones. Appendix D.1 (Table 5) shows that, on descriptions, models that watched the clip score the same as or worse than generic templates and outputs generated without the clip. This is not sufficiently discussed.
-
There are several inconsistencies in the reported numbers. Section 4.1 mentions 65 annotators and 7,176 annotations, while Appendix D reports 71 annotators in the full study and 48 in the pilot, yet a total of 83 annotators. Table 1 reports 4.6k annotations, not 7,176 as Section 4.1. The PAS values in Section 6.1 do not match those in Table 5.
-
The AUROCs for topic transition detection are very low, close to chance, and several configurations fall even below 50%. The main text does not explain why performance is so low. According to the Supplementary, even human annotators barely agree on whether a candidate is a real redirection attempt. This should be better explained in the main paper.
-
Minor issues:
-
Figure 2 contains a toilet icon between the sessions (presumably a break); please clarify.
-
There are many spelling errors
Questions for authors
-
What are the exact inputs, outputs, and metrics for Apparent Intent Inference? Please define them in the main paper.
-
What are the final numbers of annotators, annotations, and clips? Please reconcile the numbers.
-
Which PAS setup produced the values in Sec. 6.1?
Thank you to the authors for the rebuttal. Although the new explanations and corrections that are intended will improve the paper, my decision remains a weak reject.
Benchmark 3 is still unresolved: revising the labels and evaluating on human-verified samples is a good decision, but when this is done, the benchmark's conclusions will need to be reanalysed and rewritten.
I appreciate the transparency about the error that was found (human reference included comparisons between annotations written by the same annotator), but correcting this changes the reported values, and again, the results need to be re-discussed. The same goes for the new PAS protocol and normalized scores: bringing them into the main manuscript means rewriting a section entirely.
The ethical concerns were not addressed.
Building the dataset involved a lot of work and the data is valuable, but the manuscript needs substantial changes, and it should go through a new round of review.