acceptodds
Under review as a conference paper at ICLR 2027

SocialBench : Evaluating Multimodal Perception, Understanding and Reasoning in Social Settings

Abstract

Social understanding requires perceiving behavioral cues, interpreting them in context, and reasoning about implicit intentions. Evaluating only final answers offers an incomplete view of these complementary skills. We introduce SOCIALBENCH, a unified benchmark organizing 16 existing datasets into 30 task variants across three levels: Perception, Understanding, and Theory of Mind. It combines direct assessments of social cues with contextual and inferential tasks across image, video, audio, and text inputs. Evaluation of five multimodal large language models reveals strengths in selected semantic categorization tasks alongside difficulties in fine-grained spatial and temporal prediction and implicit social inference. On Qwen3-Omni, one epoch of task-specific supervised fine-tuning (SFT) substantially improves several initially weak tasks, while joint training brings further gains at the cost of regressions elsewhere. To examine how these improvements relate to internal representations, we train supervised linear predictors on activations extracted before and after SFT. These predictors outperform generated answers on several social tasks, revealing predictive information that the model’s responses do not fully reflect. After SFT, generated answers often improve more than probe predictions, narrowing but not consistently eliminating this gap. Thus, neither zero-shot nor fine-tuned responses alone fully characterize the social competencies evaluated here. SOCIALBENCH provides a framework for studying social capabilities through generated performance, supervised adaptation, and linear decodability together.

Primary area:
datasets and benchmarks
Keywords:
MLLM, Social, multimodal, LLM, Social Scenes, benchmark, evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cUt2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SOCIALBENCH organizes 16 datasets into 30 primary variants spanning social perception, understanding, and reasoning. It combines five models' zero-shot outputs with Qwen3-Omni adaptation and pre/post-training linear probes to study performance and decodable information.

Critical strengths

Generated answers, adaptation, and linear decodability provide useful complementary diagnostics.

Critical weaknesses

Main scores still conflate semantic answers with field extraction. On VocalSound, MiniCPM-o 4.5 scores 18.9% in the main table versus 62.8% under lenient extraction with the same JSON prompt; Gemma scores 29.2% versus 41.2%. Table 8 attributes this to valid JSON fields ignored by the strict scorer. Prompt-format controls do not correct the main evaluation. Please report semantic performance separately from strict compliance and revisit model comparisons under consistent extraction.

Table 1 places VideoCoAtt, EMOTIC VAD, and diarization in L2, whereas Figures 5/6 place them in L1. Figure 5 contains 32 variants including two additional CoT variants. Table 6's 29 visible rows also do not reproduce its averages: Qwen3's visible-row mean is 98.10%, versus 97% reported. Please provide consistent task IDs, levels, sample counts, variants, and aggregation weights and update the figures.

Joint-training gains are not separated from repeated exposure. Single-task training uses one epoch, while square-root joint sampling repeats small tasks; RLDD has only 96 training examples and rises from 0.480 to 0.800. Actual exposure counts and exposure-matched single-task controls are needed to identify cross-task benefits.

E.1 defines gain as ​(M_hint−M)/|M|​, but Figures 10–13 label positive values as better. Distance, AAE, and DER require direction reversal. Please reconcile the formula and plotting implementation and identify the variants behind repeated column names.

Questions for authors

as the weaknesses above.

2 comments

We thank the Reviewer for the constructive comments. We address them below.


W1. Main scores still conflate semantic answers with field extraction. [...] Please report semantic performance separately from strict compliance and revisit model comparisons under consistent extraction.

(This concern is shared with Reviewer fPms (W1), who receives the same answer.)

We agree and have revised the evaluation. All zero-shot results are recomputed from the sample-level predictions: an unreadable answer receives the naive baseline's prediction, and the native metric (e.g., F1) is computed on the full test set. This replaces the linear averaging of subset and baseline scores of the submitted Table 8.

The main results now use a lenient parser, which also accepts answers given under another key or in free text. The strict parser is also reported in the appendix.

The two parsers rank the models almost identically (Table B), and the model comparison in Sec. 4 is revised accordingly. Strict scores also change with the corrected substitution.

Table A. Share of usable answers, mean over the 30 tasks.

Qwen3-OmniQwen2.5-OmniMiniCPM-o 4.5MiniCPM-o 2.6Gemma-4 E4B
Lenient parser99.6%95.1%91.6%85.6%97.9%
Strict parser99.6%89.9%71.3%60.4%92.6%

Table B. Mean rank over the 30 tasks (1 = best; ties averaged). In parentheses: tasks where the model is best.

Qwen3-OmniQwen2.5-OmniMiniCPM-o 4.5Gemma-4 E4BMiniCPM-o 2.6
Lenient1.57 (18)3.03 (5)3.02 (4)3.67 (2)3.72 (1)
Strict1.52 (19)2.93 (5)3.25 (4)3.48 (2)3.82 (0)

Changes in the paper: Tables 3–5 and 10 (lenient and strict scores), Table 6 (usable answers per task), App. C.1 (scoring), Sec. 4, Figs. 2–3, Tables 7–8, and the probe vs. output figures (Figs. 4, 15 and 16), where the zero-shot Proxemics points move.


W2. Table 1 places VideoCoAtt, EMOTIC VAD, and diarization in L2, whereas Figures 5/6 place them in L1. [...] Please provide consistent task IDs, levels, sample counts, variants, and aggregation weights and update the figures.

We thank the Reviewer for this careful check. We added a unique ID to every task variant and corrected the paper:

  • Levels and variants. Figs. 5 and 6 are regenerated with the levels of Table 1 (VideoCoAtt, EMOTIC VAD and diarization in L2). The two exploratory variants are removed, so all tables and figures contain the same 30 variants.
  • Task IDs. Each variant has a unique ID, introduced in Table 1 and used in all tables and figures.
  • Table 6. It is recomputed like Tables 3–5 and covers the 30 variants; its last row is the unweighted mean of the rows above.
  • Sample counts and weights. Each task is evaluated on 3,000 test samples, or on the full test set when smaller; the count of each task is given in Fig. 5. All averages across tasks weight the tasks equally.

W3. Joint-training gains are not separated from repeated exposure. [...] Actual exposure counts and exposure-matched single-task controls are needed to identify cross-task benefits.

(This concern is shared with Reviewer fPms (W2), who receives the same answer.)

The results of the submitted paper already indicate that exposure does not explain the joint gains. Exposure does differ (Table 11), but its correlation with the gain is not significant (Fig. 10; Spearman ρ = +0.26, p = 0.17): under-exposed tasks also gain (VideoCoAtt localization, ×0.08), and MuStARD loses performance despite more exposure.

Nevertheless, we implemented the suggested experiments: we trained the single-task models of RLDD (×2.65) and Proxemics (×1.20) for up to three epochs, with three seeds, matching or exceeding their joint exposure. These exposure-matched controls do not reproduce the joint gains (Table C; RLDD: 25 test samples).

Table C (added as Table 12, App. D.2). Single-task SFT for k epochs (mean [min–max], 3 seeds) vs. joint SFT.

Taskk = 1k = 2k = 3Joint
Proxemics (mAP).323 [.323–.323].338 [.335–.340].327 [.324–.331].594 (×1.20)
Prox. + skel. (mAP).303 [.292–.309].312 [.300–.321].324 [.308–.336].586 (×1.20)
RLDD (Acc.).520 [.52–.52].507 [.48–.52].507 [.48–.56].800 (×2.65)

Changes in the paper: new App. D.2 with the exposure figure (Fig. 10), the exposure of every task (Table 11) and the three-epoch controls (Table 12).


W4. E.1 defines gain as (M_hint−M)/|M|, but Figures 10–13 label positive values as better. [...]

The Reviewer is right. These figures (now Figs. 11–14) were computed with the sign reversed for lower-is-better metrics; only the definition in App. E.1 omitted it. It is now corrected; the repeated columns were exploratory variants that were not kept, and all columns now use task IDs.

Thank you for the revisions and additional experiments. The corrected evaluation protocol, consistent task reporting, and exposure-matched controls address many of my concerns and provide stronger support for the results. I appreciate the authors’ efforts to improve the manuscript. I have no further questions and have raised my score to 3.

Reviewer_HrLjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SocialBench, which organizes 16 existing datasets into 30 task variants covering perception, understanding, and reasoning/Theory-of-Mind-related tasks across images, video, audio, and text. It evaluates five multimodal models using a common prompting and output protocol together with task-specific metrics. The authors further conduct single-task and joint SFT on Qwen3-Omni-30B-A3B-Thinking and compare generated-answer performance with linear decodability before and after adaptation. The results show substantial adaptation gains on several initially weak tasks, performance trade-offs under joint training, and cases where supervised linear probes outperform generated answers. Fine-tuning narrows this gap on several tasks without consistently eliminating it.

Critical strengths
  • SocialBench brings together social tasks previously studied across different research areas, covering multiple input modalities and classification, regression, spatial localization, and temporal prediction outputs. This organization provides a more differentiated assessment of social capabilities than evaluations focused on an individual task or final-answer question answering alone.

  • Beyond reporting zero-shot scores, the paper studies single-task and joint fine-tuning and compares linear decodability before and after adaptation. These analyses help distinguish performance without task-specific adaptation, improvements obtainable through supervision, and predictive information accessible through supervised linear readouts.

Critical weaknesses
  • The training and evaluation data primarily consolidate existing datasets. For a Datasets and Benchmarks submission, the contribution beyond broader task coverage and unified organization remains insufficiently articulated.

  • Single-task/joint fine-tuning and before/after-SFT probing are conducted only on Qwen3-Omni-30B-A3B-Thinking. Whether these findings extend to other model families and dense architectures remains unverified.

Questions for authors
  • Could the authors further clarify the specific contributions of this work to datasets and benchmarks, particularly its added value beyond consolidating existing datasets?

  • Could the authors add fine-tuning and linear-probing results for Qwen2.5-Omni-7B and one model from another family, such as MiniCPM-o, on representative tasks to assess the generalizability of the main findings?

2 comments

We thank the Reviewer for the constructive comments. We address them below.


W1. The training and evaluation data primarily consolidate existing datasets. For a Datasets and Benchmarks submission, the contribution beyond the broader task coverage and unified organization remains insufficiently articulated.

We respectfully disagree: the contribution of SocialBench goes beyond broader task coverage and unified organization. It not only provides a benchmark to evaluate MLLMs on a broad range of social tasks, but also a deeper analysis of where these models fall short and of their current limitations.

First, it brings these datasets into a single set of social tasks organized in three cognitive levels (perception, understanding, theory of mind), across image, video, audio and text, with one evaluation protocol, a naive baseline for every task and a published SOTA where one exists. This allows models to be compared directly across tasks and modalities, and shows where they remain far from task-specific methods.

Second, for the same tasks, it compares zero-shot answers, single-task and joint fine-tuning, and linear probes before and after fine-tuning. This allows us to identify the source of each limitation: either the information is missing from the model's representations (e.g., EMOTIC categories, MMEW), which calls for dedicated work, or it is present but not expressed in the answers (e.g., Proxemics), which mainly calls for better adaptation.

SocialBench is therefore useful both to evaluate new models and to guide future work on social understanding in MLLMs.


W2. Single-task/joint fine-tuning and before/after-SFT probing are conducted only on Qwen3-Omni-30B-A3B-Thinking. Whether these findings extend to other model families and dense architectures remains unverified.

Following the Reviewer's suggestion, we ran both on Qwen2.5-Omni-7B (dense) and on MiniCPM-o 4.5 (another family), with the protocol used for Qwen3-Omni: single-task SFT on all 30 tasks (one LoRA adapter per task, one epoch, the same hyperparameters; MEVIEW, which has no training split, uses the RLDD adapter), and the same probes on the 28 tasks probed for Qwen3-Omni. The results are added to a new appendix of the paper (App. G).

The main findings hold for both models (Table D). Single-task SFT improves 28 of the 30 tasks. Before SFT, the linear probe beats the generated answer on most tasks; after SFT, the gap narrows and the probe–output correlation increases. The gap before SFT is larger for the two smaller models (median 0.16 and 0.19 of the SOTA, against 0.07 for Qwen3-Omni), and SFT closes most of it.

Table D. Summary over tasks (26 tasks with a published SOTA for the probe columns).

SFT improvesProbe > answer (0-shot → SFT)Probe–answer r (0-shot → SFT)
Qwen3-Omni27/3019 → 160.72 → 0.84
Qwen2.5-Omni-7B28/3021 → 150.70 → 0.82
MiniCPM-o 4.528/3024 → 190.84 → 0.91

Changes in the paper: new App. G with the per-task results of both models (Tables 19–22) and their probe vs. answer figures (Figs. 19–22).


Q1. Could the authors further clarify the specific contributions of this work to datasets and benchmarks, particularly its added value beyond consolidating existing datasets?

See our answer to W1.


Q2. Could the authors add fine-tuning and linear-probing results for Qwen2.5-Omni-7B and one model from another family, such as MiniCPM-o, on representative tasks to assess the generalizability of the main findings?

Yes (SFT on all 30 tasks, probes on 28, same as Qwen3-Omni, on two more models, including dense architecture); see our answer to W2.

I thank the authors for the additional cross-model experiments, which address my concern about the generalizability of the single-task fine-tuning and linear-probing findings. However, I still think that the contribution beyond consolidating existing datasets, broadening task coverage, and unifying evaluation remains insufficiently articulated. Therefore, I will maintain my original rating.

Reviewer_fPmsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

SocialBench aggregates 16 existing datasets into 30 task variants covering social perception, contextual understanding, and proxies for social reasoning. Beyond evaluating five multimodal models, the paper studies single-task and joint LoRA adaptation of Qwen3-Omni and compares generated answers with supervised linear readouts before and after adaptation.

Critical strengths
  • The benchmark covers diverse modalities and prediction types, including classification, continuous affect estimation, spatial localization, and temporal speaker assignment.
  • Evaluating generated performance, supervised adaptability, and linear decodability together provides a useful characterization beyond zero-shot answer accuracy.
  • The paper reports heterogeneous and negative results and acknowledges important limitations, including small evaluation sets, the proxy nature of the ToM tasks, and the restricted adaptation budget.
Critical weaknesses
  • The main evaluation protocol requires revision. Appendix C.1 documents substantial differences caused by parsing: for example, MiniCPM-o 4.5 scores 18.9% on VocalSound in the main table, whereas the JSON-request condition with lenient parsing reaches 62.8% in Table 8. The authors attribute major discrepancies to valid answers stored under keys the strict metric does not read. These corrections should be incorporated consistently into the main results on matched examples. Moreover, Table 8 linearly averages recoverable-subset scores and baseline scores; for nonlinear metrics such as F1, this is not equivalent to substituting predictions and recomputing the full-set metric.
  • Joint-versus-single-task comparisons confound cross-task transfer with target-task exposure. Single-task training uses one epoch, whereas square-root joint sampling changes the effective number of repetitions substantially. With 96 RLDD training examples and batch size 16, its single-task adapter receives only approximately six updates. Exposure-matched controls are needed to distinguish transfer from additional target-task training, and interference from reduced exposure.
Questions for authors
  • Can all main results be recomputed using consistent semantic answer recovery, with native metrics calculated from complete sample-level predictions?
  • Do the joint-training gains and regressions persist against target-exposure-matched single-task controls?
  • When will the benchmark be released?
3 comments

We thank the Reviewer for the constructive comments. We address them below.


W1. The main evaluation protocol requires revision. [...] These corrections should be incorporated consistently into the main results on matched examples. Moreover, Table 8 linearly averages recoverable-subset scores and baseline scores; for nonlinear metrics such as F1, this is not equivalent to substituting predictions and recomputing the full-set metric.

(This concern is shared with Reviewer cUt2 (W1), who receives the same answer.)

We agree and have revised the evaluation. All zero-shot results are recomputed from the sample-level predictions: an unreadable answer receives the naive baseline's prediction, and the native metric (e.g., F1) is computed on the full test set. This replaces the linear averaging of subset and baseline scores of the submitted Table 8.

The main results now use a lenient parser, which also accepts answers given under another key or in free text. The strict parser is also reported in the appendix.

The two parsers rank the models almost identically (Table B), and the model comparison in Sec. 4 is revised accordingly. Strict scores also change with the corrected substitution.

Table A. Share of usable answers, mean over the 30 tasks.

Qwen3-OmniQwen2.5-OmniMiniCPM-o 4.5MiniCPM-o 2.6Gemma-4 E4B
Lenient parser99.6%95.1%91.6%85.6%97.9%
Strict parser99.6%89.9%71.3%60.4%92.6%

Table B. Mean rank over the 30 tasks (1 = best; ties averaged). In parentheses: tasks where the model is best.

Qwen3-OmniQwen2.5-OmniMiniCPM-o 4.5Gemma-4 E4BMiniCPM-o 2.6
Lenient1.57 (18)3.03 (5)3.02 (4)3.67 (2)3.72 (1)
Strict1.52 (19)2.93 (5)3.25 (4)3.48 (2)3.82 (0)

Changes in the paper: Tables 3–5 and 10 (lenient and strict scores), Table 6 (usable answers per task), App. C.1 (scoring), Sec. 4, Figs. 2–3, Tables 7–8, and the probe vs. output figures (Figs. 4, 15 and 16), where the zero-shot Proxemics points move.


W2. Joint-versus-single-task comparisons confound cross-task transfer with target-task exposure. [...] Exposure-matched controls are needed to distinguish transfer from additional target-task training, and interference from reduced exposure.

(This concern is shared with Reviewer cUt2 (W3), who receives the same answer.)

The results of the submitted paper already indicate that exposure does not explain the joint gains. Exposure does differ (Table 11), but its correlation with the gain is not significant (Fig. 10; Spearman ρ = +0.26, p = 0.17): under-exposed tasks also gain (VideoCoAtt localization, ×0.08), and MuStARD loses performance despite more exposure.

Nevertheless, we implemented the suggested experiments: we trained the single-task models of RLDD (×2.65) and Proxemics (×1.20) for up to three epochs, with three seeds, matching or exceeding their joint exposure. These exposure-matched controls do not reproduce the joint gains (Table C; RLDD: 25 test samples).

Table C (added as Table 12, App. D.2). Single-task SFT for k epochs (mean [min–max], 3 seeds) vs. joint SFT.

Taskk = 1k = 2k = 3Joint
Proxemics (mAP).323 [.323–.323].338 [.335–.340].327 [.324–.331].594 (×1.20)
Prox. + skel. (mAP).303 [.292–.309].312 [.300–.321].324 [.308–.336].586 (×1.20)
RLDD (Acc.).520 [.52–.52].507 [.48–.52].507 [.48–.56].800 (×2.65)

Changes in the paper: new App. D.2 with the exposure figure (Fig. 10), the exposure of every task (Table 11) and the three-epoch controls (Table 12).


Q1. Can all main results be recomputed using consistent semantic answer recovery [...]?

Yes, see our answer to W1.


Q2. Do the joint-training gains and regressions persist against target-exposure-matched single-task controls?

For the gains, yes (see W2). The regressions are not explained by reduced exposure either: 22 of the 29 tasks are under-exposed, and among them only 10 regress while 10 improve.


Q3. When will the benchmark be released?

The anonymized repository linked in the submitted paper already contains the scripts to build the data splits and to run the evaluation to benchmark other models. The final version of the repository, released publicly as soon as anonymity is lifted, will also include our training scripts.

Thanks for the revisions and additional experiments. The corrected scoring protocol addresses my evaluation concerns, and the longer single-task runs provide stronger support for the joint-training gains.

One remaining question is whether reduced target-task exposure could explain some of the performance drops. Improvements on other under-exposed tasks do not rule this out for the tasks that regress. Could you add an exposure-matched single-task comparison for a representative regressing task, or acknowledge this uncertainty when discussing negative transfer?

We thank the Reviewer for the follow-up and are glad that the revised scoring and longer single-task runs address the evaluation concerns.

We agree that gains on other under-exposed tasks do not rule out an exposure effect on the regressing tasks. Following the Reviewer's suggestion, we started exposure-matched single-task runs for three of them: GazeFollow (×0.05 exposure), VideoCoAtt detection (×0.04) and AffWild2 AU detection (×0.28), each trained on as many target samples as the joint model sees (Table 11). These could not be completed within the discussion period; we will add them as three rows to Table 12 in the final version, and state this uncertainty when discussing negative transfer in Section 5.

Comments on the paper

2 comments

We thank the Reviewers for their constructive comments. In the revised paper that we uploaded, we clarified several points of the evaluation and added the experiments requested by the Reviewers. All changes are shown in magenta, and the captions of the new figures and tables in the appendix start with [New].

These revisions do not change the conclusions of the paper: both parsers rank the models almost identically, the joint-training gains are not explained by additional target-task exposure, and the adaptation and probing findings hold on two other models.

Two overlapping concerns were raised independently by two reviewers:

  • Format compliance in the zero-shot scores (fPms W1, cUt2 W1).
  • Joint-training gains and target-task exposure (fPms W2, cUt2 W3).

To streamline the reading process, the same response is provided in both answer boxes below.

Main changes and their location in the paper:

  • Zero-shot scoring. All zero-shot results are recomputed from sample-level predictions, with an exact per-sample substitution for unusable answers. The main results use a lenient parser (answer correctness), and strict scores (format compliance) are reported alongside. See Sec. 4, Fig. 2, Tables 3–6 and 10, and App. C.1. The discussion of Sec. 4 is revised accordingly, with a new paragraph on answer correctness and format compliance. The output-format ablation uses the same scoring (Tables 7–8, Fig. 7).
  • Joint vs. single-task training. We checked that the gains of joint training are not simply due to the model seeing some tasks more often. We report how many training samples of each task the joint model sees, and, for the two tasks where this could matter (RLDD and Proxemics), we trained the single-task model for up to three epochs, which does not reproduce the joint gains (App. D.2, Fig. 10, Tables 11–12).
  • Other models. Single-task SFT and linear probing are repeated on Qwen2.5-Omni-7B and MiniCPM-o 4.5 (new App. G, Tables 19–22, Figs. 19–22; Sec. 6 refers to it).
  • Consistency. Every task variant has a unique ID, used in all tables and figures (Table 1). Figs. 5–6 follow the levels of Table 1 and contain the 30 variants. Table 6 is recomputed. Metric directions, small-test-set markers and units are harmonized across tables.
  • Cross-task gain. The definition now accounts for lower-is-better metrics (App. E.1).

Dear Reviewers,

The authors have responded to your questions. If you haven’t already done so, please review their responses and let them know whether your concerns have been addressed. Please use the discussion to raise any remaining concerns or follow-up questions.

Thank you for your time!

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.