acceptodds
Under review as a conference paper at ICLR 2027

Social Micro-Strategy Understanding from Multimodal Behavior

Abstract

Understanding social interaction requires not only recognizing people's emotions, intentions, and beliefs, but also interpreting how people influence, maintain, and adjust ongoing interactions. However, existing social understanding benchmarks largely focus on mental states or coarse-grained behaviors, leaving fine-grained interactional functions and their multimodal behavioral cues underexplored. We introduce a structured framework for social micro-strategies that characterizes fine-grained interactional functions and grounds their interpretation in observable multimodal behavior. Based on this framework, we construct SMSBench, a benchmark spanning text, audio, and visual modalities, with detailed annotations across diverse social scenarios. A comprehensive evaluation of state-of-the-art multimodal large language models reveals substantial deficiencies in social micro-strategy understanding. We further fine-tune a 7B model using the training data to investigate task-specific adaptation. Experimental results show that non-expert humans slightly outperform current models overall, while fine-tuning yields clear improvements. We will release our dataset and code upon acceptance.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Social Micro-Strategies, Multimodal Large Language Models, Evidence Grounding, Human Interaction, Benchmark Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LQnbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the gap that existing social understanding benchmarks mostly focus on mental states or coarse-grained behaviors rather than fine-grained interactional functions. It proposes a hierarchical framework of social micro-strategies — structured interactional units that regulate ongoing social interactions — and constructs SMSBench, a large-scale multimodal benchmark grounded in observable verbal and non-verbal behavioral evidence.

Key contributions:

  1. A layered taxonomy of interactional functions (7 primary + 27 fine-grained) paired with five channels of multimodal evidence (text, prosody, face, body, target response).
  2. SMSBench: 996 source videos, 4,051 interaction segments, and 5,600 annotated strategy instances across 7 real-world social scenarios, with quality-controlled annotations.
  3. A systematic evaluation framework covering strategy recognition and evidence grounding, with extensive empirical analysis.

Validation findings:

Current multimodal LLMs show substantial limitations in complete strategy recovery, fine-grained function recognition, and non-textual evidence grounding. Supervised fine-tuning yields large performance gains, while reinforcement learning provides only marginal improvements. Non-expert human performance only slightly exceeds the strongest closed-source model. Modality ablations reveal complementary contributions across channels, and strategy recognition and evidence grounding are strongly correlated across models but only partially coupled at the event level.

Critical strengths
  1. Systematic benchmark framework with diagnostic evaluation

The paper does not merely add another social‑understanding dataset; it establishes a complete paradigm of hierarchical interaction‑function taxonomy paired with multimodal evidence grounding, advancing social AI evaluation from coarse mental‑state classification to fine‑grained interactional function analysis. The benchmark is built with rigorous multi‑stage quality control, and the evaluation covers 13 mainstream MLLMs with systematic ablation studies (modality, oracle information, sampling rate, adaptation methods, human baseline) that clearly map out capability boundaries and bottlenecks rather than just ranking models. This infrastructural contribution enables a wide range of follow‑up research and is the primary reason to accept the paper. 2. Novel conceptual framing of social micro‑strategies

The notion of “social micro‑strategy” — focusing on what interactional function a behavior serves, rather than what emotion or intent it expresses — addresses a pervasive blind spot in existing multimodal social understanding benchmarks. The dual‑task design (strategy recognition + evidence grounding) also differs from standard classification benchmarks and better mirrors the full process of real social understanding. This conceptual and methodological novelty gives the work lasting relevance beyond its immediate benchmark results.

Critical weaknesses
  1. Moderate annotation reliability and severe class imbalance in fine-grained categories

The fine-grained interactional function (FIF) annotations achieve only moderate inter-annotator reliability (Krippendorff’s α = 0.473), indicating notable subjectivity inherent to the micro-strategy taxonomy. In addition, the category distribution is heavily long-tailed: many FIF classes have very sparse samples, so performance estimates for these fine-grained functions lack statistical robustness. This directly weakens the core novelty of the paper — the hierarchical micro-strategy framework — and reduces confidence in the fine-grained model evaluation conclusions. 2. Primary evaluation relies on strong oracle settings that simplify the task

The main benchmark task operates on pre-segmented interaction clips and provides the gold number of strategy instances per segment, both of which are oracle priors unavailable in real end-to-end social understanding scenarios. Ablation results show that removing the gold strategy count substantially degrades overall performance and halves the segment-level exact match rate, indicating the main evaluation setting overestimates models’ real-world capability. This limits the benchmark’s practical utility as a measure of natural social understanding performance.

Questions for authors

The reported inter-annotator reliability for Fine-grained Interactional Functions (FIF) is moderate (Krippendorff’s α = 0.473), and many FIF categories have very sparse sample sizes. Could the authors clarify whether statistical confidence intervals or significance tests are provided for per-class performance estimates, especially for low-frequency categories? Additionally, is there additional validity evidence supporting the discriminability of the fine-grained taxonomy? The primary benchmark operates on pre-segmented clips and provides the gold number of strategy instances per segment, both of which are oracle inputs not available in real-world end-to-end social understanding. Ablation results show that removing the gold strategy count substantially degrades performance. Could the authors explain why this strong-oracle setting was chosen as the primary evaluation rather than a more natural end-to-end setup, and whether they have plans to include end-to-end (unknown count, unsegmented) results as a complementary benchmark?

Reviewer_Git9OpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SMSBench, a multimodal benchmark for understanding fine-grained social interaction strategies. It defines a hierarchical taxonomy of interactional functions and constructs a dataset of 996 videos containing 5,600 annotated strategy instances. The benchmark evaluates structured Actor–Target–Function recognition and multimodal evidence grounding. Experiments with various MLLMs reveal difficulties in recognizing fine-grained strategies and grounding them in behavioral evidence.

Critical strengths
  1. The paper addresses an interesting problem beyond conventional emotion and intent recognition. Representing social interactions through Actor–Target relations and hierarchical functions is a useful direction.

  2. SMSBench contains diverse interaction scenarios and detailed multimodal annotations. The evaluation includes several recent MLLMs, modality ablations, human comparisons, and task-specific adaptation.

Critical weaknesses
  1. The proposed task is not sufficiently distinguished from existing work on dialogue acts, communicative intent, and multimodal social reasoning. The paper should clarify what capabilities SMSBench uniquely evaluates.

  2. The annotation reliability raises concerns. The reported Krippendorff's alpha is 0.591 for PIF and only 0.473 for FIF, indicating considerable ambiguity in the fine-grained taxonomy.

  3. The evaluation simplifies strategy recognition by providing the ground-truth number of strategy instances. Evidence grounding is further conditioned on Gold strategy labels.

  4. The evidence grounding evaluation relies on an LLM judge comparing predictions with annotated textual evidence, without directly inspecting the original video or audio.

  5. The modality comparison is not fully controlled across models with different audio and visual capabilities.

Questions for authors
  1. Clarify the distinction from existing social intent and dialogue-act benchmarks.

  2. Address the relatively low agreement for fine-grained labels.

  3. Clarify the limitations of Gold-conditioned evaluation.

  4. Provide stronger validation of the multimodal grounding scores.

2 comments
Area_Chair_PRJ8OpenReview ↗

Please follow up before the discussion period ends & AI use clarification

Dear reviewer,

We are less than a day away from the end of the discussion period. Please read the authors' response and any revisions, and post a brief follow-up indicating which concerns remain and whether your assessment has changed. Please raise any outstanding questions promptly so the authors have time to respond.

Given the limited time, please focus on clarifications and existing evidence. You may identify gaps in the experimental support, but please do not expect the authors to conduct new experiments before the discussion period closes.

Additionally, your review was flagged for AI use while the corresponding field in your review does not mention how it was used. Please clarify this for me.

Thank you for your contribution!

Best,

Your AC

Dear Area Chair,

Thank you for the reminder. I’ve updated the AI documentation field in my review. The authors clarified what the agreement figures measure. My remaining concern is that the judge compares written descriptions without checking the cue against the original clip. I’m keeping my rating.

Best,

Reviewer

Reviewer_YBZjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SMSBench, a video benchmark for understanding fine-grained social interaction strategies from multimodal human behavior. The authors propose a hierarchical taxonomy of primary interactional functions (PIFs) and fine-grained interactional functions (FIFs), and provides annotations on these functions on 996 YouTube collected videos, together with the actor-target relationships and supporting evidence. The authors conducted evaluations across mainstream open and closed source models, with the strongest model achieving 42.34% overall accuracy, compared with 44.34% for non-expert humans. The authors also experimented with fine-tuning methods including supervised fine tuning and reinforcement learning, and found that they are effective in improving model performance on SMSBench.

Critical strengths
  • the benchmark emphasizes on fine-grained behavior and strategy within a social interaction, which is novel compared to prior social social video QA benchmarks.
  • the dataset construction required substantial human effort, with detailed annotation on the interaction functions and reasoning evidence. The evaluation is comprehensive as well, including open-source and close-source models, SFT and RL finetuning, and modality ablations.
Critical weaknesses
  • while the proposed social micro-strategy framework is intuitive, the authors did not justify why the categories were chosen and how they are justified by social theory literature. This makes the hierarchy looks rather ad-hoc at the moment.
  • the benchmark itself might not be be challenging enough, given that Gemini-3.1-Pro already approaches human performance (or even surpasses on some metrics). Although the authors claim that the human performance resembles non-expert humans, they did not provide evidence on whether how "expert humans" would perform on this benchmark.
  • while the benchmark focuses on multimodal social behavior, the results from the modality ablation indicates that multimodality might be less important in this benchmark. For example, text-only overall performance is only about 3% less than vision+audio+text performance, raising concern whether the multimodal social cues are actually needed, or are there linguistic biases or shortcuts in the benchmark's question-answer constructions.
Questions for authors
  • could the authors provide a clearer justification of the micro-strategy framework, especially how they are derived from the social interaction theories literature?
  • given that Gemini-3.1-Pro already approaches non-expert human performance and surpasses it on several metrics, could the authors provide further evidence of the benchmark's difficulty, and show that higher human performance can be achieved?
  • could the authors provide an interpretation on the relatively small performance improvement from text-only to full multimodal input on this benchmark, and explain whether whether nonverbal cues are genuinely necessary for solving the benchmark?
1 comment
Area_Chair_PRJ8OpenReview ↗

Please follow up before the discussion period ends

Dear reviewer,

We are less than a day away from the end of the discussion period. Please read the authors' response and any revisions, and post a brief follow-up indicating which concerns remain and whether your assessment has changed. Please raise any outstanding questions promptly so the authors have time to respond.

Given the limited time, please focus on clarifications and existing evidence. You may identify gaps in the experimental support, but please do not expect the authors to conduct new experiments before the discussion period closes.

Thank you for your contribution!

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.