acceptodds
Under review as a conference paper at ICLR 2027

SocialWorldBench: Benchmarking Social Understanding and Prediction

Abstract

Can AI understand a conversation and predict how it will end? We introduce SocialWorldBench, a benchmark of 19,107 questions drawn from six existing datasets. Its main test uses 396 human negotiation dialogues. For each dialogue, a model reads the same conversation excerpt and answers two questions: what negotiation strategies are being used, and how satisfied will a participant be afterward? We compare the satisfaction prediction with what the participant later reported. This lets researchers examine whether models that recognize negotiation strategies also make accurate predictions. We test five large language models from one provider on a 1,401-question sample and the paired negotiation task. When scored by answer-label overlap (Jaccard), three models identify strategies worse than a simple method that always gives the most common answer. Changing the evaluation questions also reverses two models’ rankings, even though their answers stay unchanged. SocialWorldBench provides researchers with an evaluation dataset to test whether AI can understand conversations and predict their outcomes.

Primary area:
datasets and benchmarks
Keywords:
social world models, social reasoning, behavior prediction, large language models, theory of mind, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Gu7qOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces SocialWorldBench, a benchmark that integrates 19,107 questions from six existing datasets to evaluate social understanding and prediction. Its main contribution is a paired evaluation framework that examines whether models can both understand social interactions and predict subsequent outcomes. The paper also introduces dataset auditing procedures and an action-conditioned forecasting framework.

The authors evaluate five LLMs from a single provider on a subset of 1,401 questions, along with a paired evaluation using 396 negotiation dialogues. The experiments reveal variations in model performance across tasks and evaluation metrics. However, several proposed experiments remain unexecuted, including LLM evaluation on the full Dynamics track. As a result, the paper provides limited empirical evidence supporting the effectiveness of the proposed benchmark for evaluating social forecasting capabilities.

Critical strengths

None

Critical weaknesses

1. Serious concerns about the completeness and overall quality of the submission.

The manuscript reads more like an internal research log than a completed scientific paper. Several particularly concerning examples include:

  • Empty figures (Figures 1 and 2, pp. 33 and 63): The paper includes two numbered figures containing no actual experimental results. Their captions explicitly state that the corresponding analyses were never conducted and that no data are available to plot. I find it difficult to understand why such placeholder figures were retained in a conference submission.

  • Detailed API budget bookkeeping (Appendix S.13, p. 60): The authors devote space to explaining that a model with a 39.2% error rate was discarded and could not be rerun because only \28.12remainedfroma28.12 remained from a \\205 budget, while the rerun would cost \$35.16. This level of internal expense tracking is highly unusual in a scientific paper and contributes nothing to its scientific findings.

  • Unsupported human-performance references (Appendix S.11, p. 59): The authors acknowledge that earlier versions used human-performance anchors that were either arbitrary placeholders or had no documented provenance. Some were supposedly derived from a calibration dataset that was never constructed. Although these values were later withdrawn, their initial inclusion raises serious concerns about the research's quality control.

  • Largely invalidated human evaluation (Appendix T.4, p. 63): Of the 140 items in the original human evaluation, only 16 remained unchanged in the final benchmark. Consequently, the authors cannot provide a valid human-performance reference for the released evaluation.

  • Extensive documentation of unexecuted experiments (Appendix C, pp. 19–20): The paper dedicates multiple pages to a detailed human-interaction experiment protocol, including scenarios, randomization, recording procedures, and outcome measures, despite explicitly stating that no data were collected.

These are only some examples of a broader problem throughout the manuscript. Rather than presenting a coherent scientific contribution supported by completed experiments, the paper repeatedly documents failed attempts, withdrawn results, unfinished analyses, and explanations for missing evidence. The writing is excessively verbose and defensive, often resembling a preemptive rebuttal rather than a research paper.

I find the overall lack of scientific quality control and manuscript preparation deeply concerning. In its current form, the submission falls substantially below the standard of completeness and rigor expected of an ICLR conference paper.

Questions for authors

I have no specific questions for the authors. My concerns are not about unclear explanations or missing clarifications, but rather fundamental issues with the scientific rigor, completeness, and overall quality of the submission. I do not believe these concerns can be adequately addressed through a rebuttal.

Reviewer_9CMAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper releases SocialWorldBench, 19,107 items re-packaged from six existing corpora (CaSiNo, FANToM, OpenToM, SOTOPIA, MELD, Social-IQ 2.0), with a shortcut audit, quarantine of solvable heads, and checksummed item lists. Its main experiment pairs a static question (negotiation strategy) with a forecasting question (self-reported satisfaction) on 396 CaSiNo dialogues, and reports that across five Claude models the two rankings correlate at ρ = 0.80, with three models below the majority-class baseline on the static half.

Critical strengths

none

Critical weaknesses
  1. The main experiments are missing. No model was tested on the 1,600 prediction questions, which are the point of the benchmark. Simple checks, such as removing the conversation history, were never run, and the action-prediction part tests no language model at all.

  2. The main finding rests on too little evidence: one dataset of 396 dialogues and only five models, all from one company. The correlation could be anywhere from 0.11 to 1.00, and none of the ranking differences is statistically significant.

  3. Mainly an engineering and compilation effort. No new data is collected, and the questions are reformatted from six existing datasets, and the paired-question idea and the checking methods come from earlier work.

  4. No clear results in the main paper. There are no figures, the only results table shows just the lowest and highest score per dataset, and the per-model results for the main experiment appear only on page 78.

  5. The released data does not match the paper. The data link is titled "NegotiationBench", was updated after submission, and disagrees with the paper (e.g., it calls a subset human-checked when the paper says it is not, and releases non-commercial-only data under a licence that allows commercial use).

  6. Unreliable process. The code used made-up human scores with no source (e.g., 0.82 for MELD and 0.78 for CaSiNo, L3144–3147, two more at L3173–3176), and the LLM-run literature review described three of five cited results wrongly in earlier drafts (L1701–1705, L2786–2787). Both were found only shortly before submission, which raises doubt about what else was missed. The code is also not released (L476–478).]

  7. No human comparison and no overall score. All human reference scores were removed, and only 16 of the 140 questions rated by people (two of the authors) remain in the final benchmark.

  8. Many questions are of doubtful quality. 71% come from stories written by an LLM, video and audio datasets are given to models as text only, and the authors' own raters could not decide the right answer for about 75% of MELD questions from the text.

  9. Very hard to read. The paper is 80 pages, introduces many new terms, repeats the same warnings and long sentences again and again, and spends most pages describing work that was not done.

  10. Feels unfinished. Some sentences are cut off midway, some figures are empty placeholders, and different appendices contradict each other.

  11. Poor table formatting. Six tables go past the page margin, Table 13 runs off the page so one column is cut off, and the caption of Table 12 runs off the bottom of page 78.

  12. Does not use the font recommended by the ICLR template (Times), and the title, headings and bold/italic text lose their styling.

Questions for authors
  1. Can you test at least one model on the prediction questions, and with the conversation history removed?

  2. Does the main result hold on another dataset, or with models from other companies?

  3. Why are the per-model results for the main experiment not in the main paper?

  4. Why is the released dataset called "NegotiationBench", and why does it disagree with the paper?

Reviewer_9hefOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SocialWorldBench, a benchmark consisting of 396 human negotiation dialogues that evaluates the conversational understanding of AI models regarding (1) identifying the negotiation strategies being used, and (2) the extent of participant satisfaction.

Critical strengths

None

Critical weaknesses
  1. The writing and presentation can be further polished. For example, the submission uses the wrong LaTeX format (such as font style and margins).
  2. This paper seems overclaimed, while the evaluation experiments are not extensive, and many key results and analyses are in the appendix.
Questions for authors
  1. The proposed SocialWorldBench claims to benchmark social understanding and prediction, but in the implementation, each instance asks, "What negotiation strategies are being used, and how satisfied will a participant be afterward?" However, it does not seem to cover "social understanding and prediction" well.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.