acceptodds
Withdrawn from ICLR 2027

E-MMSI: Toward Evidence-Grounded Social Interaction Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce Evidence-grounded Multi-party Multi-modal Social Interaction (E-MMSI) understanding, a new task for spoken social videos. Beyond answering an interaction question, a model must provide observable evidence that supports its answer, making the prediction verifiable. To study this problem systematically, we draw on interaction theory to build an evidence benchmark on existing social-interaction datasets. The annotation schema represents answer-relevant actors, activities, partners, relations, contexts, and temporal anchors. Evaluation shows that current models often produce unreliable evidence in complex and dynamic multi-party interactions, such as grounding a pointing gesture to the wrong participant or time interval. To address this challenge, we construct a word-level social representation (WSR) that captures dynamic verbal and non-verbal social cues in word interval. Averaged across the three benchmark subsets, WSR improves answer accuracy over the baseline by 3.47 points and evidence score by 13.01 points.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal social interaction understanding, evidence grounding, multimodal large language models, video understanding, social reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZKKTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Evidence-grounded Multi-party Multi-modal Social Interaction (E-MMSI), a task requiring models to answer social-interaction questions and provide structured supporting evidence specifying actor, activity, temporal anchor, partner, relation, and context. With this benchmark, this paper evaluates some current close- and open-source models and finds that current models fail to provide reliable evidence even when providing accurate answer. To help deal with this challenge, this paper proposes a word-aligned social representation (WSR), a heuristic method which uses spoken-word intervals as shared temporal anchors for verbal cues and concurrent non-verbal observations. Experiments in this paper demonstrate that WSR improves both answer accuracy and evidence grounding.

Critical strengths
  1. The task addresses a meaningful gap. E-MMSI is a benchmark requiring accurate answer and evidence with five interaction elements. This fine-grained evaluation can help distinguish models' correct answer with incorrect evidence.
  2. The method is helpful. External WSR consistently improve both answer accuracy and evidence grounding.
Critical weaknesses
  1. The task addresses a meaningful gap. E-MMSI is a benchmark requiring accurate answer and evidence with five interaction elements. This fine-grained evaluation can help distinguish models' correct answer with incorrect evidence.
  2. The method is helpful. External WSR consistently improve both answer accuracy and evidence grounding.

Critical Weaknesses

  1. The completeness of reference evidence remains insufficiently validated. Social-IQ 2.0 exhibits relatively low inter-annotator agreement (56.95 Evidence F1), which the authors partly attribute to annotators prioritizing different valid supporting cues. Although the independent video-grounded audit reports high annotation quality (95.92%), it only checks whether annotated elements are supported by the video. This raises the possibility that the reference annotations do not cover all valid evidence for a question. Consequently, the reference-based Evidence F1 may penalize valid alternative evidence generated by models.
  2. Limited coverage of proprietary model families. Although the benchmark includes multiple open-source backbones, its proprietary-model evaluation is restricted to the Claude family.
  3. WSR’s generalization to proprietary models remains untested. The paper demonstrates WSR’s benefits on several open-source backbones but provides no controlled comparison of a proprietary model with and without WSR.
  4. Potentially unfair comparison. Appendix E states that Claude models design and invoke their own media-processing tools whose pipeline is different from other baselines'. This makes the comparisons with other baselines potentially unfair. What's more, tool-assisted Claude evaluation is not mentioned in 5.1 Setup, which make it hard to review.
  5. The scope of WSR’s benefits need stronger validation. In Table 5, Qwen3-VL-32B + WSR matches the Whisper baseline’s accuracy but reduces Evidence F1 from 46.24 to 40.73. In Table 13, +Whisper outperforms +WSR in both answer accuracy and overall Evidence F1 for all three evaluated backbones on Social-IQ 2.0. These results indicate substantial backbone and setting dependence.
Questions for authors
  1. Why are the Claude models evaluated using self-designed media-processing tools rather than the same preprocessing pipeline as the other models?
  2. It seems that WSR does not consistently improve performance on Social-IQ 2.0. Although the authors suggest that smaller backbones may explain some degradations, Qwen3-VL-32B also achieves lower Evidence F1 with WSR than with Whisper alone. Could you provide further evidence of WSR’s effectiveness and clarify which model characteristics or experimental settings determine when it helps or hurts performance?
  3. WSR relies on several additional tools for transcription, speaker identification, tracking, and keypoint estimation. What computational and latency overhead do these tools introduce, and how does the end-to-end cost compare with the baseline methods?
Reviewer_CCZROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces E-MMSI, a benchmark of 1,006 social-video questions with structured supporting evidence covering participants, activities, temporal intervals, relationships, and context. It proposes Evidence F1 for element-level evaluation and a training-free word-aligned social representation (WSR) combining speaker-attributed transcripts with visual overlays. Experiments across three task subsets and multiple backbones report improvements in answer accuracy and evidence scores.

Critical strengths
  • The benchmark separates answer correctness from evidence quality and provides useful diagnostics of participant attribution, interaction targets, and timing. This addresses limitations of accuracy-only evaluation in social-video understanding.

  • WSR improves reported overall evidence scores across the three main backbones. Component ablations and alternative-judge evaluation provide useful, though limited, evidence of complementary inputs and robustness.

Critical weaknesses
  • The contribution of word-level alignment is not isolated. WSR simultaneously adds transcription, speaker attribution, temporal information, and visual overlays. Existing work already explores dense multimodal alignment (MMSI), socially aware visual prompting (Online-MMSI), and tool-based identity attribution (Omni-MMSI). Without information-matched alignment-granularity ablations or controlled comparisons with these approaches, the experiments establish the benefit of the combined pipeline more clearly than the proposed mechanism. Moreover, WSR is not consistently better than transcription alone: in Table 5, Qwen3-VL + Whisper achieves 46.24 Evidence F1 versus 40.73 with WSR, at identical answer accuracy.

  • The interpretation and comparability of the evaluation require stronger support. Evidence F1 measures reference agreement rather than directly verifying audiovisual correctness or whether evidence supports the predicted answer. Its temporal-only matching permits an incorrect activity to receive credit for the other fields; a single four-field event can score 75 despite a completely incorrect activity. Additionally, several Table 5 accuracies are incompatible with ordinary per-question accuracy over the stated 719 questions. These issues limit confidence in both the meaning of the evidence improvements and cross-model comparisons.

Questions for authors
  • What exact sample sets, denominators, and aggregation rules were used for each Table 5 model? For example, 87.50% cannot result from integer correct counts over 719 questions, while 82.69%, 87.50%, and 85.58% are compatible with 86/104, 91/104, and 89/104. Were all models evaluated on identical questions, and how were failures handled?

  • Do you have comparisons between word-level and utterance-level or fixed-window alignment that retain identical transcripts, participant identities, visual cues, and comparable input budgets? Such results would materially strengthen attribution of the gains to word-level alignment.

  • Have model evidence scores been validated against independent human judgments of video-grounded correctness and answer support? How sensitive are the main gains to removing temporal-only matches or requiring activity agreement before awarding event credit?

Reviewer_6LMpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper says that for social videos, a model should not just answer the question but also point to the evidence: who said or did what, to whom, and when. It introduces E-MMSI, a benchmark for this task, built by re-annotating about 1,000 questions from Omni-MMSI and Social-IQ 2.0 with structured evidence. Predictions are scored with an Evidence F1 that matches claims element by element.

It also proposes WSR, a representation built from off-the-shelf tools. Speech is transcribed with word timestamps and speaker labels, and frames get overlays for participant boxes, gaze, and hands, all lined up on the word timeline. The model receives the video, the question, and WSR, and regenerates only if it outputs an invalid time interval. WSR improves both accuracy and Evidence F1 on the three main backbones, and the two halves of WSR work best together.

Critical strengths
  • The task makes sense and the evidence schema is simple. Breaking scores down by element shows clearly where models fail, and the finding that timing and partner of non-verbal cues are the hardest parts is useful.
  • The annotation process is careful: trained annotators, an expert in the loop, model-flagged review, agreement numbers, and a separate audit against the video.
  • Many models are evaluated, and the WSR gains on the two Omni-MMSI tasks hold for every open backbone tested.
  • The authors are honest about what does not work, including cases where WSR hurts and the judge-sensitivity check.
  • The qualitative examples are good and show the kind of error the benchmark is designed to catch.
Critical weaknesses
  1. The baseline is too weak for the vision-only backbones. For the two VLLMs, the headline gains are measured against base models that get no speech at all. Against the same models plus a Whisper transcript, the evidence gains are much smaller, and for Qwen3-VL-32B on Social-IQ 2.0 WSR is even worse than Whisper.
  2. Word-level alignment is never tested by itself. The only coarser baseline is a plain Whisper transcript without speaker labels. Also, frames are sampled at 1 FPS, and most words are shorter than that.
  3. The benchmark is small and there are no error bars. PCR has only 112 questions, so many differences in the tables are one or two questions. All results are single runs, and the human baseline uses only 10 questions per subset. The refinement step also changes almost nothing.
Questions for authors
  1. . With 1 FPS sampling, how are frames assigned to words, and what does the non-verbal text channel contain?
  2. Was thinking mode off for Qwen3-Omni-Thinking in all tables, or only in the ablation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.