acceptodds
Under review as a conference paper at ICLR 2027

GESBENCH: When and Where Do Multimodal Language Models Perceive Social Meaning?

Abstract

Human social perception is incremental: we infer intent before an utterance is complete, revise our interpretation as new cues arrive, and attend to specific facial, gestural, or prosodic signals that shape social meaning. Current multimodal LLM benchmarks largely evaluate final-answer understanding over complete episodes, rewarding post-hoc recognition of social meaning rather than the unfolding perception required for real-time interaction. Drawing on interactionally embedded Gestalt principles of multimodal human communication, we introduce GesBench: a human-grounded benchmark for cue-level social perception in multimodal LLMs. GesBench uses cumulative sub-utterance clips to measure when models form stable interpretations of speaker intention and social affordance, and human gaze-guided corruption to evaluate whether regions attended by humans are functionally important for model judgments. Built from more than utterance-level clips from scripted and unscripted conversational datasets, GesBench includes human annotations of unfolding judgments and gaze on a representative subset. Across open and closed multimodal models, we find a clear human-model gap: humans form predictive understandings early and revise more often, while models often hedge or repeat across different clips. GesBench shifts social evaluation from whether models eventually recognize meaning to when and where that meaning is formed. Our codes are publicly available at https://anonymous.4open.science/status/GesBench-share-212C.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Gestalt perception, MLLM evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LRMhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces GESBENCH, a novel benchmark for evaluating social perception in MLLMs. Unlike existing benchmarks focused on post-hoc recognition, GESBENCH draws on Gestalt principles to assess when and where social meaning emerges during conversations.

Critical strengths
  1. Interesting Perspective: This paper introduces an interesting perspective on evaluating social perception in MLLMs by drawing on Gestalt principles, shifting the focus from post-hoc recognition to incremental social understanding.

  2. Comprehensive Evaluation: GESBENCH attempts to evaluate social perception from multiple perspectives, including temporal anticipation, interpretation revision, and spatial sensitivity to visual cues.

  3. Human-grounded Annotations: The authors collect human annotations of evolving social interpretations and eye-gaze data, providing a valuable human reference for evaluating and comparing model behaviors.

  4. Extensive Experiments: The experiments cover multiple conversational datasets, including both scripted and unscripted interactions, and evaluate a range of open- and closed-source MLLMs, providing insights into their different social perception behaviors.

Critical weaknesses

Evaluation Metrics

  1. Gestalt Perception: This paper proposes a benchmark grounded in Gestalt Perception. While I acknowledge that the proposed metrics may capture certain aspects of Gestalt Perception, I believe further clarification or validation is needed to establish whether these individual measurements are sufficient to evaluate how models integrate multimodal cues to construct coherent social meaning.

  2. Specificity: As I understand it, Specificity measures the semantic distinctiveness of responses across different videos at comparable temporal stages, assigning higher scores when responses are less similar. However, I am not fully convinced that this metric adequately captures video-specific social perception, as it appears to measure cross-video response distinctiveness rather than actual understanding of the video content. In particular, generating different responses for different videos does not necessarily indicate that the model correctly interprets their social meaning. More fundamentally, even if responses were randomly reassigned across videos, the overall Specificity score might remain largely unchanged, since response diversity would be preserved. Conversely, a model providing accurate but similar interpretations for videos depicting the same social intention could receive a lower score.

  3. Revision Types (RT): While RT provides an interesting perspective on how interpretations evolve over time, I am not fully convinced that more frequent revisions necessarily indicate better social perception. A model that correctly identifies the speaker's intention early and maintains a consistent interpretation could exhibit fewer revisions, whereas a model that frequently changes its interpretation due to uncertainty or incorrect predictions could exhibit a higher revision rate.

Experiments

  1. This paper emphasizes Gestalt Perception, which involves integrating multimodal cues to construct social meaning. However, the Temporal Evaluation only reports results using both video and audio inputs, making it difficult to determine whether the models' early predictions and interpretation revisions arise from multimodal integration or primarily rely on audio or linguistic information. While I acknowledge that Appendix C.6 includes GMS experiments without audio, I believe further analysis of the contribution of each modality in the Temporal Evaluation would help strengthen the paper's claims.
Questions for authors

Evaluation Metrics

1.1 RT: Could the authors further clarify why a higher frequency of interpretation changes should be considered indicative of more human-like or effective social perception? Although Appendix C.5 examines the value of updating, it remains unclear whether individual revisions are justified by newly observed evidence.

1.2 RT: Have the authors validated the NLI-based RT classifications against human annotations? Such validation would help establish the reliability of the proposed metric, particularly in distinguishing genuine interpretation changes from semantic variations in responses.

  1. Since the Temporal and Spatial Pipelines are evaluated independently, I wonder whether this approach sufficiently captures the relationship between when social meaning is formed and where the relevant visual cues are located. For example, analyzing which visual cues contribute to changes in the model's interpretation at specific moments could further strengthen the paper's claims regarding Gestalt Perception.

Experiments

1.(Minor Question) I am curious why GPT-series models were not included in the evaluation. Could the authors clarify the reason for their exclusion?

  1. Human Gaze-based Masking: I am curious about the additional evaluation value of human gaze-based masking compared to conventional visual region masking. In the current experiments, the observed changes in model responses might simply result from masking important visual regions, such as faces, which humans naturally tend to attend to. Therefore, I believe that additional comparisons with random masking or face-centered masking, while controlling for the masked area, would help clarify whether the observed effects are specifically attributable to human gaze patterns.

Overall

  1. I'm curious about the motivation for evaluating human-like Gestalt Perception in MLLMs. While I understand the importance of incremental social perception in real-time interactions, could the authors further clarify why human-like perception patterns are desirable for AI models? In particular, is there any evidence or theoretical justification suggesting that closer alignment with human perception leads to better social interaction capabilities in AI agents?
Reviewer_a7QKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors curate a dataset aimed at evaluating the capability of making social-perception judgments during human utterances, as opposed to one single judgment after an entire episode. They also use gaze measurements to corrupt clips in particular places to test whether human importance aligns with machine importance in this task. They evaluate several MLLMs and show that humans decide earlier, and revise more often whereas models reserve judgment.

Critical strengths
  • The paper tackles a highly important problem, which I believe is currently under-studied. While most social video understanding tasks and benchmarks evaluate judgments made after watching an entire utterance or conversation, GesBench investigates the small decisions made during tiny sub-utterance intervals, which is well-motivated by Gestalt theory.
  • The dataset provided is likely to be a useful resource to the community
  • The results are interesting and informative: models tend to reserve judgment until the end, whereas humans revise their judgments more frequently. There is also an interesting finding that only the Qwen2.5 model was sensitive to the corruption of human-gaze locations
Critical weaknesses

My main concerns are about the soundness, and whether the results align with the paper's claims.

  • The abstract emphasizes over 6000 utterances, but from what I understand, only 600 carry your human annotations, and the remaining 5400 have no additional annotations
  • The way that humans and the models provide judgments may be the thing causing the main difference: humans speak their answers and are told not to verbatim repeat. Models see their previous answers and say if their perception changed yes/no. This seems to bias the models to "copying", which is one of your main results.
  • For Specificity, the difference could easily be due to models saying boilerplate phrases like "the speaker seems to be trying to...", which would reduce the specificity for models. First cleaning the model answers of the boilerplate phrases would provide a much more fair comparison with humans.
  • Perhaps the biggest concern is the human judgments, whose quality we do not know. Because you do not compare these judgments with previous annotations, and do not report inter-annotator agreement, we have no way of knowing whether the annotations are just random noise or something that nearly any annotator would agree with.
Questions for authors
  • What was the final-answer agreement between annotators? You could compare this to a cross-video baseline.
  • The link to the anonymous repo did not load (stuck at 0%), so I was not able to view any real samples.
  • I am open to increasing my score, but my main concern is the quality of the human judgments and of the task itself, which can only really be alleviated if I am shown statistics that show it is a consistent task with consistent annotators.
Reviewer_keMHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GESBench evaluates when and where multimodal models form social interpretations. It combines cumulative video clips with human gaze-guided masking to assess early specificity, anticipation, interpretation revision, and corruption sensitivity. Experiments reveal that humans generally form more specific early interpretations and revise them more frequently, while models vary considerably in their sensitivity to human-attended regions.

Critical strengths
  • The focus on incremental social understanding addresses a meaningful limitation of complete-video evaluation. Measuring how interpretations develop provides useful information beyond final-answer performance.

  • The combination of human response trajectories and eye-tracking data provides a valuable empirical reference. The appendix’s update-gain analysis strengthens the temporal findings by checking whether revisions improve agreement with human interpretations.

Critical weaknesses
  • The temporal metrics may conflate perceptual ability with response style and prompting effects. Embedding similarity and automatic NLI classification do not directly establish interpretation correctness, and supplying previous answers may encourage repetition. The extraction and length checks are useful but do not fully resolve these concerns.

  • The spatial results have limited explanatory power: low sensitivity can indicate either robustness or limited visual use. Additionally, Appendix B.3 allows masks to be clipped at frame boundaries, potentially changing the actual occluded area across offsets and confounding the location comparison.

Questions for authors
  • Metric validation: Was revision classification validated against human judgments on a representative sample of both human and model responses? Such agreement estimates would help establish the reliability of the reported revision gap.

  • Prompt sensitivity: Does the temporal gap persist when previous answers are omitted or when response format and length are controlled?

  • Spatial controls: How much does the actual masked area vary across offsets, and does the gaze-location effect persist when visible mask area is matched?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.