- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the gap that existing social understanding benchmarks mostly focus on mental states or coarse-grained behaviors rather than fine-grained interactional functions. It proposes a hierarchical framework of social micro-strategies — structured interactional units that regulate ongoing social interactions — and constructs SMSBench, a large-scale multimodal benchmark grounded in observable verbal and non-verbal behavioral evidence.
Key contributions:
- A layered taxonomy of interactional functions (7 primary + 27 fine-grained) paired with five channels of multimodal evidence (text, prosody, face, body, target response).
- SMSBench: 996 source videos, 4,051 interaction segments, and 5,600 annotated strategy instances across 7 real-world social scenarios, with quality-controlled annotations.
- A systematic evaluation framework covering strategy recognition and evidence grounding, with extensive empirical analysis.
Validation findings:
Current multimodal LLMs show substantial limitations in complete strategy recovery, fine-grained function recognition, and non-textual evidence grounding. Supervised fine-tuning yields large performance gains, while reinforcement learning provides only marginal improvements. Non-expert human performance only slightly exceeds the strongest closed-source model. Modality ablations reveal complementary contributions across channels, and strategy recognition and evidence grounding are strongly correlated across models but only partially coupled at the event level.
Critical strengths
- Systematic benchmark framework with diagnostic evaluation
The paper does not merely add another social‑understanding dataset; it establishes a complete paradigm of hierarchical interaction‑function taxonomy paired with multimodal evidence grounding, advancing social AI evaluation from coarse mental‑state classification to fine‑grained interactional function analysis. The benchmark is built with rigorous multi‑stage quality control, and the evaluation covers 13 mainstream MLLMs with systematic ablation studies (modality, oracle information, sampling rate, adaptation methods, human baseline) that clearly map out capability boundaries and bottlenecks rather than just ranking models. This infrastructural contribution enables a wide range of follow‑up research and is the primary reason to accept the paper. 2. Novel conceptual framing of social micro‑strategies
The notion of “social micro‑strategy” — focusing on what interactional function a behavior serves, rather than what emotion or intent it expresses — addresses a pervasive blind spot in existing multimodal social understanding benchmarks. The dual‑task design (strategy recognition + evidence grounding) also differs from standard classification benchmarks and better mirrors the full process of real social understanding. This conceptual and methodological novelty gives the work lasting relevance beyond its immediate benchmark results.
Critical weaknesses
- Moderate annotation reliability and severe class imbalance in fine-grained categories
The fine-grained interactional function (FIF) annotations achieve only moderate inter-annotator reliability (Krippendorff’s α = 0.473), indicating notable subjectivity inherent to the micro-strategy taxonomy. In addition, the category distribution is heavily long-tailed: many FIF classes have very sparse samples, so performance estimates for these fine-grained functions lack statistical robustness. This directly weakens the core novelty of the paper — the hierarchical micro-strategy framework — and reduces confidence in the fine-grained model evaluation conclusions. 2. Primary evaluation relies on strong oracle settings that simplify the task
The main benchmark task operates on pre-segmented interaction clips and provides the gold number of strategy instances per segment, both of which are oracle priors unavailable in real end-to-end social understanding scenarios. Ablation results show that removing the gold strategy count substantially degrades overall performance and halves the segment-level exact match rate, indicating the main evaluation setting overestimates models’ real-world capability. This limits the benchmark’s practical utility as a measure of natural social understanding performance.
Questions for authors
The reported inter-annotator reliability for Fine-grained Interactional Functions (FIF) is moderate (Krippendorff’s α = 0.473), and many FIF categories have very sparse sample sizes. Could the authors clarify whether statistical confidence intervals or significance tests are provided for per-class performance estimates, especially for low-frequency categories? Additionally, is there additional validity evidence supporting the discriminability of the fine-grained taxonomy? The primary benchmark operates on pre-segmented clips and provides the gold number of strategy instances per segment, both of which are oracle inputs not available in real-world end-to-end social understanding. Ablation results show that removing the gold strategy count substantially degrades performance. Could the authors explain why this strong-oracle setting was chosen as the primary evaluation rather than a more natural end-to-end setup, and whether they have plans to include end-to-end (unknown count, unsegmented) results as a complementary benchmark?
Please follow up before the discussion period ends & AI use clarification
Dear reviewer,
We are less than a day away from the end of the discussion period. Please read the authors' response and any revisions, and post a brief follow-up indicating which concerns remain and whether your assessment has changed. Please raise any outstanding questions promptly so the authors have time to respond.
Given the limited time, please focus on clarifications and existing evidence. You may identify gaps in the experimental support, but please do not expect the authors to conduct new experiments before the discussion period closes.
Additionally, your review was flagged for AI use while the corresponding field in your review does not mention how it was used. Please clarify this for me.
Thank you for your contribution!
Best,
Your AC