- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces SocialWorldBench, a benchmark that integrates 19,107 questions from six existing datasets to evaluate social understanding and prediction. Its main contribution is a paired evaluation framework that examines whether models can both understand social interactions and predict subsequent outcomes. The paper also introduces dataset auditing procedures and an action-conditioned forecasting framework.
The authors evaluate five LLMs from a single provider on a subset of 1,401 questions, along with a paired evaluation using 396 negotiation dialogues. The experiments reveal variations in model performance across tasks and evaluation metrics. However, several proposed experiments remain unexecuted, including LLM evaluation on the full Dynamics track. As a result, the paper provides limited empirical evidence supporting the effectiveness of the proposed benchmark for evaluating social forecasting capabilities.
Critical strengths
None
Critical weaknesses
1. Serious concerns about the completeness and overall quality of the submission.
The manuscript reads more like an internal research log than a completed scientific paper. Several particularly concerning examples include:
-
Empty figures (Figures 1 and 2, pp. 33 and 63): The paper includes two numbered figures containing no actual experimental results. Their captions explicitly state that the corresponding analyses were never conducted and that no data are available to plot. I find it difficult to understand why such placeholder figures were retained in a conference submission.
-
Detailed API budget bookkeeping (Appendix S.13, p. 60): The authors devote space to explaining that a model with a 39.2% error rate was discarded and could not be rerun because only \205 budget, while the rerun would cost \$35.16. This level of internal expense tracking is highly unusual in a scientific paper and contributes nothing to its scientific findings.
-
Unsupported human-performance references (Appendix S.11, p. 59): The authors acknowledge that earlier versions used human-performance anchors that were either arbitrary placeholders or had no documented provenance. Some were supposedly derived from a calibration dataset that was never constructed. Although these values were later withdrawn, their initial inclusion raises serious concerns about the research's quality control.
-
Largely invalidated human evaluation (Appendix T.4, p. 63): Of the 140 items in the original human evaluation, only 16 remained unchanged in the final benchmark. Consequently, the authors cannot provide a valid human-performance reference for the released evaluation.
-
Extensive documentation of unexecuted experiments (Appendix C, pp. 19–20): The paper dedicates multiple pages to a detailed human-interaction experiment protocol, including scenarios, randomization, recording procedures, and outcome measures, despite explicitly stating that no data were collected.
These are only some examples of a broader problem throughout the manuscript. Rather than presenting a coherent scientific contribution supported by completed experiments, the paper repeatedly documents failed attempts, withdrawn results, unfinished analyses, and explanations for missing evidence. The writing is excessively verbose and defensive, often resembling a preemptive rebuttal rather than a research paper.
I find the overall lack of scientific quality control and manuscript preparation deeply concerning. In its current form, the submission falls substantially below the standard of completeness and rigor expected of an ICLR conference paper.
Questions for authors
I have no specific questions for the authors. My concerns are not about unclear explanations or missing clarifications, but rather fundamental issues with the scientific rigor, completeness, and overall quality of the submission. I do not believe these concerns can be adequately addressed through a rebuttal.