- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SOCIALBENCH organizes 16 datasets into 30 primary variants spanning social perception, understanding, and reasoning. It combines five models' zero-shot outputs with Qwen3-Omni adaptation and pre/post-training linear probes to study performance and decodable information.
Critical strengths
Generated answers, adaptation, and linear decodability provide useful complementary diagnostics.
Critical weaknesses
Main scores still conflate semantic answers with field extraction. On VocalSound, MiniCPM-o 4.5 scores 18.9% in the main table versus 62.8% under lenient extraction with the same JSON prompt; Gemma scores 29.2% versus 41.2%. Table 8 attributes this to valid JSON fields ignored by the strict scorer. Prompt-format controls do not correct the main evaluation. Please report semantic performance separately from strict compliance and revisit model comparisons under consistent extraction.
Table 1 places VideoCoAtt, EMOTIC VAD, and diarization in L2, whereas Figures 5/6 place them in L1. Figure 5 contains 32 variants including two additional CoT variants. Table 6's 29 visible rows also do not reproduce its averages: Qwen3's visible-row mean is 98.10%, versus 97% reported. Please provide consistent task IDs, levels, sample counts, variants, and aggregation weights and update the figures.
Joint-training gains are not separated from repeated exposure. Single-task training uses one epoch, while square-root joint sampling repeats small tasks; RLDD has only 96 training examples and rises from 0.480 to 0.800. Actual exposure counts and exposure-matched single-task controls are needed to identify cross-task benefits.
E.1 defines gain as (M_hint−M)/|M|, but Figures 10–13 label positive values as better. Distance, AAE, and DER require direction reversal. Please reconcile the formula and plotting implementation and identify the variants behind repeated column names.
Questions for authors
as the weaknesses above.
2 comments
Thank you for the revisions and additional experiments. The corrected evaluation protocol, consistent task reporting, and exposure-matched controls address many of my concerns and provide stronger support for the results. I appreciate the authors’ efforts to improve the manuscript. I have no further questions and have raised my score to 3.
We thank the Reviewer for the constructive comments. We address them below.
(This concern is shared with Reviewer fPms (W1), who receives the same answer.)
We agree and have revised the evaluation. All zero-shot results are recomputed from the sample-level predictions: an unreadable answer receives the naive baseline's prediction, and the native metric (e.g., F1) is computed on the full test set. This replaces the linear averaging of subset and baseline scores of the submitted Table 8.
The main results now use a lenient parser, which also accepts answers given under another key or in free text. The strict parser is also reported in the appendix.
The two parsers rank the models almost identically (Table B), and the model comparison in Sec. 4 is revised accordingly. Strict scores also change with the corrected substitution.
Table A. Share of usable answers, mean over the 30 tasks.
Table B. Mean rank over the 30 tasks (1 = best; ties averaged). In parentheses: tasks where the model is best.
Changes in the paper: Tables 3–5 and 10 (lenient and strict scores), Table 6 (usable answers per task), App. C.1 (scoring), Sec. 4, Figs. 2–3, Tables 7–8, and the probe vs. output figures (Figs. 4, 15 and 16), where the zero-shot Proxemics points move.
We thank the Reviewer for this careful check. We added a unique ID to every task variant and corrected the paper:
(This concern is shared with Reviewer fPms (W2), who receives the same answer.)
The results of the submitted paper already indicate that exposure does not explain the joint gains. Exposure does differ (Table 11), but its correlation with the gain is not significant (Fig. 10; Spearman ρ = +0.26, p = 0.17): under-exposed tasks also gain (VideoCoAtt localization, ×0.08), and MuStARD loses performance despite more exposure.
Nevertheless, we implemented the suggested experiments: we trained the single-task models of RLDD (×2.65) and Proxemics (×1.20) for up to three epochs, with three seeds, matching or exceeding their joint exposure. These exposure-matched controls do not reproduce the joint gains (Table C; RLDD: 25 test samples).
Table C (added as Table 12, App. D.2). Single-task SFT for k epochs (mean [min–max], 3 seeds) vs. joint SFT.
Changes in the paper: new App. D.2 with the exposure figure (Fig. 10), the exposure of every task (Table 11) and the three-epoch controls (Table 12).
The Reviewer is right. These figures (now Figs. 11–14) were computed with the sign reversed for lower-is-better metrics; only the definition in App. E.1 omitted it. It is now corrected; the repeated columns were exploratory variants that were not kept, and all columns now use task IDs.