acceptodds
Under review as a conference paper at ICLR 2027

COSI-Lab: Conference Living Lab for Modeling Multi-Perspective Multimodal Social Intention

Abstract

COSI-Lab presents a multimodal multi-sensor dataset of a real interdisciplinary scientific workshop with 32 academics at an international conference. It captures ecologically valid social interactions in a weakly scripted setting consisting of two 30-minute mingling sessions with real professional and social consequences. We argue that intelligent systems could better handle subjective perceptions by modeling their multiplicity not as label noise but as an explainable perspective-driven reasoning process. We focus on the Apparent Intent Inference (AII) problem as determined by ex-situ observers and conceptualize intentions to be independent of the future. We contribute a novel annotation process for AII that accounts for a perceiver's own interpretative tendencies, quantitative and qualitative analyses of intent narratives with respect to diversity, plausibility and grounding, benchmark tasks for AII and relevant contextual factors, speech quality audio for all participants as well as privacy-preserving multi-modal environmental and wearable sensor data, enabling lexical and nonverbal behavior analysis; and the coupling of self-reported participant goals (30 minute to 3 hour) with annotated AII (seconds).

Primary area:
datasets and benchmarks
Keywords:
social intention, explanation, subjectivity, multi-modal, multi-sensor, audio-visual, MLLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zkQ3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a multimodal dataset for studying social intentions in natural interactions, along with three tasks defined on it: apparent intention estimation, social involvement identification, and topic changes.

The data were recorded during a real workshop at an international conference with 32 participants, and it comprises two 30-minute mingling sessions in which participants moved and talked freely, without assigned partners or topics. Participants completed personality and demographic surveys beforehand, and reported their goals and mood before each session and whether they achieved them afterwards. Annotators, who also completed personality questionnaires, watched 30-second clips and described what a given participant seemed to be trying to achieve (the description) and why (the explanation). The aim is to keep multiple interpretations per moment and relate them to each annotator's profile.

Critical strengths

The data come from a real networking event and include several synchronized modalities (video, individual audio, wearable sensors). Participants reported their own goals before and after each session, and a follow-up survey three months later recorded which connections they maintained. Annotators, who also filled in personality questionnaires, described what they believed a participant's intention was and why.

To the best of my knowledge, no existing dataset combines all of this. Such data is expensive and difficult to collect. This is the main reason I see the paper as worth considering despite its weaknesses.

Critical weaknesses
  1. The paper is hard to follow: the reader has to go back and forth between the main text and the supplementary material to understand the tasks and results.

  2. The introduction presents many theoretical concepts, but finally it does not present a concrete task definition. The central task, Apparent Intent Inference, is not clearly specified in the main paper: what is the input, what is the output, and what are the metrics?

  3. The proposed PAS metric does not distinguish clip-grounded narratives from generic ones. Appendix D.1 (Table 5) shows that, on descriptions, models that watched the clip score the same as or worse than generic templates and outputs generated without the clip. This is not sufficiently discussed.

  4. There are several inconsistencies in the reported numbers. Section 4.1 mentions 65 annotators and 7,176 annotations, while Appendix D reports 71 annotators in the full study and 48 in the pilot, yet a total of 83 annotators. Table 1 reports 4.6k annotations, not 7,176 as Section 4.1. The PAS values in Section 6.1 do not match those in Table 5.

  5. The AUROCs for topic transition detection are very low, close to chance, and several configurations fall even below 50%. The main text does not explain why performance is so low. According to the Supplementary, even human annotators barely agree on whether a candidate is a real redirection attempt. This should be better explained in the main paper.

  6. Minor issues:

  • Figure 2 contains a toilet icon between the sessions (presumably a break); please clarify.

  • There are many spelling errors

Questions for authors
  1. What are the exact inputs, outputs, and metrics for Apparent Intent Inference? Please define them in the main paper.

  2. What are the final numbers of annotators, annotations, and clips? Please reconcile the numbers.

  3. Which PAS setup produced the values in Sec. 6.1?

Thank you to the authors for the rebuttal. Although the new explanations and corrections that are intended will improve the paper, my decision remains a weak reject.

Benchmark 3 is still unresolved: revising the labels and evaluating on human-verified samples is a good decision, but when this is done, the benchmark's conclusions will need to be reanalysed and rewritten.

I appreciate the transparency about the error that was found (human reference included comparisons between annotations written by the same annotator), but correcting this changes the reported values, and again, the results need to be re-discussed. The same goes for the new PAS protocol and normalized scores: bringing them into the main manuscript means rewriting a section entirely.

The ethical concerns were not addressed.

Building the dataset involved a lot of work and the data is valuable, but the manuscript needs substantial changes, and it should go through a new round of review.

Reviewer_mNDdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents COSI-Lab, a dataset of 32 academics interacting during two 30-minute mingling sessions at a scientific workshop. It includes synchronized overhead video, individual speech recordings, wearable sensors, participant surveys, and later reports of social connections. The authors study how observers interpret what someone is trying to achieve. They collect several interpretations of each clip, together with explanations, assumptions, confidence, alternative interpretations, and annotator profiles. Their PINE framework evaluates whether model narratives are grounded in the recordings and cover the range of human interpretations. It uses the Persona Alignment Score, or PAS, to measure similarity to human narratives. Experiments compare multimodal models with human annotations, test the relationship between annotator profiles and narratives, and provide baselines for detecting conversation groups and topic transitions.

Critical strengths
  1. The dataset records interactions that matter to the participants' professional lives. Sections 3 and 4 and Appendices A and B describe how speech recordings, nonverbal behavior, participant goals, and later reports of social connections are collected together. Collecting these data is difficult. I see particular value in studying how behavior during a conversation relates to what participants hope to achieve.

  2. The annotations preserve differences between observers. Observers explain their interpretations, state their assumptions, and describe alternatives. Their responses are linked to their profiles. Section 5 uses crossed random effects and permutations of whole survey profiles to test the association between profiles and narrative similarity. This gives some evidence that the differences are worth studying, and the annotations let researchers examine how observers reach different conclusions from the same clip.

Critical weaknesses
  1. I am not convinced that the topic-transition labels are reliable enough for Benchmark 3. Tables 8 and 9 in Appendix E.4.2, on pages 42 and 43, show low human agreement on whether a candidate is a valid redirection attempt. Krippendorff's alpha is 0.094 for validity and -0.170 for success or failure among the eligible outcome cases. Against the adjudicated validity labels, the LLM has 0.424 accuracy and 0.388 macro-F1. It accepts 19 of 22 invalid candidates. The 0.909 outcome accuracy emphasized in Section 5 is measured on just 11 cases already judged valid by humans. The 0.800 attempt accuracy uses only 15 cases where the human annotators agreed. These selected cases do not establish the quality of the labels used in Figure 4B and Benchmark 3. The validation sample is small and stratified, so it cannot give an error rate for the full dataset. Still, the errors it reveals need to be addressed before I can trust the conclusions drawn from these labels.

  2. The appendix undermines the main text's interpretation of PAS. Appendix D.1 discusses this on pages 29 to 31. In Table 5, generic descriptions score 0.397, almost matching Qwen's 0.402. Persona outputs generated without seeing the clip score 0.438, above Gemma's 0.368 with access to the clip. Under the appendix's protocol, subtracting the random-clip baseline reverses the description ranking. Normalized PAS is 0.86 for Gemma and 0.73 for Qwen, and the explanation ranking is not robust. Section 6.1 nevertheless emphasizes Qwen's higher raw PAS. I appreciate that the authors report these controls, but the main comparison needs to reflect what they show. A higher score can come from generic phrasing without better understanding of the clip. The supplementary plausibility judge only reads other annotations, so it does not replace a check by humans watching the clips.

Questions for authors
  1. Were the labels used in Figure 4B and Benchmark 3 corrected after the human validation? How well does the complete labeling pipeline agree with independent human adjudication, including invalid and unresolved candidates? Do the benchmark conclusions hold on human-verified labels?

  2. Which PAS protocol should be used for the main comparison, given the different conclusions in Section 6.1 and Appendix D.1? Do the model ranking and coverage findings hold after accounting for the generic and blind controls? Are there human judgments made while viewing the clips that support the scores, particularly for less common interpretations?

Reviewer_KShYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces COSI-Lab, a dataset of 32 people, mostly researchers, who mingled in two 30-minute sessions at an academic workshop. It includes overhead video, close-talk speech with transcripts, wearable sensor data, and surveys on the participants’ goals and traits. Crowd workers watched short clips, described the intentions they saw, and reported their own traits. The authors also propose a way to evaluate model-generated intention descriptions and report three benchmarks: intention description, conversation-group detection, and topic-change detection.

Critical strengths

S1

The combination of data modalities is novel. Previous mingling datasets, such as SALSA, MatchNMingle, and ConfLab, lack clear speech recordings for each participant and do not include participants' goals or intention annotations. COSI-Lab brings these together, allowing researchers to study the relationship between speech and movement, as well as how participants' intentions are perceived by others.

S2

The annotation design is well aligned with the research question. Multiple annotators describe each clip, and their own traits are also collected. This allows researchers to examine how different observers interpret the same behavior.

Critical weaknesses

W1

The dataset is small, with 32 participants across two 30-minute mingling sessions at a single event. This is comparable in size to older mingling datasets such as SALSA, ConfLab, and MatchNMingle, but much smaller than large datasets of natural behavior with speech, such as Ego4D and Seamless Interaction. Since the data comes from a single event and one group of participants, it is unclear whether the findings generalize to other settings.

W2.

Several collected data streams, including indoor positions, the sensor on the drinking glass, and participants' trait surveys, are not used in any analysis or benchmark. The authors leave their use to future work. As a result, the paper does not demonstrate the usefulness of these data.

W3

The paper does not clearly state what is new compared with ConfLab. ConfLab introduced the conference living lab as a data collection concept and described it as a template for later editions. COSI-Lab uses the same type of event and extends the same capture setup. However, the introduction presents the living lab design as part of the contribution, and ConfLab is missing from Table 1. The main additions are speech, goal surveys, and intent annotations, and the paper should present them as extensions of ConfLab.

Questions for authors

Q1 Do you have result that needs the new combination of data? For example, a task where speech and body signals together, or stated goals and perceived intentions together, give something that neither gives alone.

Q2 The abstract and the conclusion present the link between stated goals and perceived intentions as a main contribution, but Section 5 reports only how often annotated intentions match a participant’s stated goals. How did you decide that an intention matches a goal? What is the match rate when the goals come from a random other participant, as a chance level?

Q3 The topic-change labels come from an LLM, but the human check in the appendix confirms only about a third of the cases the LLM marked as valid, and the main text does not mention this. How much can we trust these labels? And since the best result is close to chance, what does this benchmark actually show?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.