Social Micro-Strategy Understanding from Multimodal Behavior

Understanding social interaction requires not only recognizing people's emotions, intentions, and beliefs, but also interpreting how people influence, maintain, and adjust ongoing interactions. However, existing social understanding benchmarks largely focus on mental states or coarse-grained behaviors, leaving fine-grained interactional functions and their multimodal behavioral cues underexplored. We introduce a structured framework for social micro-strategies that characterizes fine-grained interactional functions and grounds their interpretation in observable multimodal behavior. Based on this framework, we construct SMSBench, a benchmark spanning text, audio, and visual modalities, with detailed annotations across diverse social scenarios. A comprehensive evaluation of state-of-the-art multimodal large language models reveals substantial deficiencies in social micro-strategy understanding. We further fine-tune a 7B model using the training data to investigate task-specific adaptation. Experimental results show that non-expert humans slightly outperform current models overall, while fine-tuning yields clear improvements. We will release our dataset and code upon acceptance.