- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces GESBENCH, a novel benchmark for evaluating social perception in MLLMs. Unlike existing benchmarks focused on post-hoc recognition, GESBENCH draws on Gestalt principles to assess when and where social meaning emerges during conversations.
Critical strengths
-
Interesting Perspective: This paper introduces an interesting perspective on evaluating social perception in MLLMs by drawing on Gestalt principles, shifting the focus from post-hoc recognition to incremental social understanding.
-
Comprehensive Evaluation: GESBENCH attempts to evaluate social perception from multiple perspectives, including temporal anticipation, interpretation revision, and spatial sensitivity to visual cues.
-
Human-grounded Annotations: The authors collect human annotations of evolving social interpretations and eye-gaze data, providing a valuable human reference for evaluating and comparing model behaviors.
-
Extensive Experiments: The experiments cover multiple conversational datasets, including both scripted and unscripted interactions, and evaluate a range of open- and closed-source MLLMs, providing insights into their different social perception behaviors.
Critical weaknesses
Evaluation Metrics
-
Gestalt Perception: This paper proposes a benchmark grounded in Gestalt Perception. While I acknowledge that the proposed metrics may capture certain aspects of Gestalt Perception, I believe further clarification or validation is needed to establish whether these individual measurements are sufficient to evaluate how models integrate multimodal cues to construct coherent social meaning.
-
Specificity: As I understand it, Specificity measures the semantic distinctiveness of responses across different videos at comparable temporal stages, assigning higher scores when responses are less similar. However, I am not fully convinced that this metric adequately captures video-specific social perception, as it appears to measure cross-video response distinctiveness rather than actual understanding of the video content. In particular, generating different responses for different videos does not necessarily indicate that the model correctly interprets their social meaning. More fundamentally, even if responses were randomly reassigned across videos, the overall Specificity score might remain largely unchanged, since response diversity would be preserved. Conversely, a model providing accurate but similar interpretations for videos depicting the same social intention could receive a lower score.
-
Revision Types (RT): While RT provides an interesting perspective on how interpretations evolve over time, I am not fully convinced that more frequent revisions necessarily indicate better social perception. A model that correctly identifies the speaker's intention early and maintains a consistent interpretation could exhibit fewer revisions, whereas a model that frequently changes its interpretation due to uncertainty or incorrect predictions could exhibit a higher revision rate.
Experiments
- This paper emphasizes Gestalt Perception, which involves integrating multimodal cues to construct social meaning. However, the Temporal Evaluation only reports results using both video and audio inputs, making it difficult to determine whether the models' early predictions and interpretation revisions arise from multimodal integration or primarily rely on audio or linguistic information. While I acknowledge that Appendix C.6 includes GMS experiments without audio, I believe further analysis of the contribution of each modality in the Temporal Evaluation would help strengthen the paper's claims.
Questions for authors
Evaluation Metrics
1.1 RT: Could the authors further clarify why a higher frequency of interpretation changes should be considered indicative of more human-like or effective social perception? Although Appendix C.5 examines the value of updating, it remains unclear whether individual revisions are justified by newly observed evidence.
1.2 RT: Have the authors validated the NLI-based RT classifications against human annotations? Such validation would help establish the reliability of the proposed metric, particularly in distinguishing genuine interpretation changes from semantic variations in responses.
- Since the Temporal and Spatial Pipelines are evaluated independently, I wonder whether this approach sufficiently captures the relationship between when social meaning is formed and where the relevant visual cues are located. For example, analyzing which visual cues contribute to changes in the model's interpretation at specific moments could further strengthen the paper's claims regarding Gestalt Perception.
Experiments
1.(Minor Question) I am curious why GPT-series models were not included in the evaluation. Could the authors clarify the reason for their exclusion?
- Human Gaze-based Masking: I am curious about the additional evaluation value of human gaze-based masking compared to conventional visual region masking. In the current experiments, the observed changes in model responses might simply result from masking important visual regions, such as faces, which humans naturally tend to attend to. Therefore, I believe that additional comparisons with random masking or face-centered masking, while controlling for the masked area, would help clarify whether the observed effects are specifically attributable to human gaze patterns.
Overall
- I'm curious about the motivation for evaluating human-like Gestalt Perception in MLLMs. While I understand the importance of incremental social perception in real-time interactions, could the authors further clarify why human-like perception patterns are desirable for AI models? In particular, is there any evidence or theoretical justification suggesting that closer alignment with human perception leads to better social interaction capabilities in AI agents?