- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Evidence-grounded Multi-party Multi-modal Social Interaction (E-MMSI), a task requiring models to answer social-interaction questions and provide structured supporting evidence specifying actor, activity, temporal anchor, partner, relation, and context. With this benchmark, this paper evaluates some current close- and open-source models and finds that current models fail to provide reliable evidence even when providing accurate answer. To help deal with this challenge, this paper proposes a word-aligned social representation (WSR), a heuristic method which uses spoken-word intervals as shared temporal anchors for verbal cues and concurrent non-verbal observations. Experiments in this paper demonstrate that WSR improves both answer accuracy and evidence grounding.
Critical strengths
- The task addresses a meaningful gap. E-MMSI is a benchmark requiring accurate answer and evidence with five interaction elements. This fine-grained evaluation can help distinguish models' correct answer with incorrect evidence.
- The method is helpful. External WSR consistently improve both answer accuracy and evidence grounding.
Critical weaknesses
- The task addresses a meaningful gap. E-MMSI is a benchmark requiring accurate answer and evidence with five interaction elements. This fine-grained evaluation can help distinguish models' correct answer with incorrect evidence.
- The method is helpful. External WSR consistently improve both answer accuracy and evidence grounding.
Critical Weaknesses
- The completeness of reference evidence remains insufficiently validated. Social-IQ 2.0 exhibits relatively low inter-annotator agreement (56.95 Evidence F1), which the authors partly attribute to annotators prioritizing different valid supporting cues. Although the independent video-grounded audit reports high annotation quality (95.92%), it only checks whether annotated elements are supported by the video. This raises the possibility that the reference annotations do not cover all valid evidence for a question. Consequently, the reference-based Evidence F1 may penalize valid alternative evidence generated by models.
- Limited coverage of proprietary model families. Although the benchmark includes multiple open-source backbones, its proprietary-model evaluation is restricted to the Claude family.
- WSR’s generalization to proprietary models remains untested. The paper demonstrates WSR’s benefits on several open-source backbones but provides no controlled comparison of a proprietary model with and without WSR.
- Potentially unfair comparison. Appendix E states that Claude models design and invoke their own media-processing tools whose pipeline is different from other baselines'. This makes the comparisons with other baselines potentially unfair. What's more, tool-assisted Claude evaluation is not mentioned in 5.1 Setup, which make it hard to review.
- The scope of WSR’s benefits need stronger validation. In Table 5, Qwen3-VL-32B + WSR matches the Whisper baseline’s accuracy but reduces Evidence F1 from 46.24 to 40.73. In Table 13, +Whisper outperforms +WSR in both answer accuracy and overall Evidence F1 for all three evaluated backbones on Social-IQ 2.0. These results indicate substantial backbone and setting dependence.
Questions for authors
- Why are the Claude models evaluated using self-designed media-processing tools rather than the same preprocessing pipeline as the other models?
- It seems that WSR does not consistently improve performance on Social-IQ 2.0. Although the authors suggest that smaller backbones may explain some degradations, Qwen3-VL-32B also achieves lower Evidence F1 with WSR than with Whisper alone. Could you provide further evidence of WSR’s effectiveness and clarify which model characteristics or experimental settings determine when it helps or hurts performance?
- WSR relies on several additional tools for transcription, speaker identification, tracking, and keypoint estimation. What computational and latency overhead do these tools introduce, and how does the end-to-end cost compare with the baseline methods?