- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work introduces SOCIALSIM BENCH, an executable multi agent benchmark covering social media and commercial negotiation scenarios. It adopts rule based metrics and five dimensional LLM as a Judge to evaluate agents’ adaptive social strategies. It further builds a simulation to decision pipeline generating conditional decision reports. Experiments reveal strong LLMs mostly use simple local strategies, and trajectory summarization improves report quality.
Critical strengths
- It designs a controlled executable benchmark with deterministic state transitions for multi agent social simulation, combining objective rule based outcomes and LLM as a Judge to measure both task results and interactive behavioral quality.
- It investigates a simulation to decision workflow, studying how simulation traces can support conditional decision analysis.
Critical weaknesses
- simulation grounded reports do not consistently outperform well structured scenario only reports, limiting practical decision support value.
- Skill and refinement interventions cannot stably improve agent performance across models and domains, lacking reliable generalizable enhancement for social agents.
Questions for authors
- 1In Section 3.3, the social media rule score in this paper adopts priority-weighted average. What is the basis for selecting the weight ? Why is weight ablation validation not carried out?
- The experiment only repeats evaluation rounds and does not use independent scenarios for ablation. Will this cause overfitting of evaluation results and reduce the reliability of the conclusion?
- This paper does not conduct direct comparison experiments with published social simulation benchmarks in the same field such as Sotopia and SocialBench. How does SOCIALSIM-BENCH reflect its unique technical advantages in evaluation dimensions compared with existing benchmarks?
- "access to raw simulation artifacts alone is insufficient", this conclusion comes from two harness baselines: Codex and Claude Code. Does selecting only these two baselines have baseline selection bias?
- The environment adopts fully deterministic state transition . Will a fully deterministic simulation environment reduce the evaluation validity of the generalization ability of real LLM social agents?
- In the reference section, a considerable portion of the literature are only preprints. It is suggested to select the latest published literature from authoritative journals in the recent three years as much as possible.
4 comments
Response to Q1. How were the weights for the social-media rule score determined? Why was there no weight ablation?
Uniform weights were used by default, with each objective assigned a weight of 1. The additional sensitivity analysis is as follows. The results are based on 36 matched episodes, and the rule-objective coverage is 100%. Under the default setting, the five deterministic objectives use equal weights, with each objective assigned before normalization. In each sensitivity setting, only one objective's weight is doubled and the other four remain at their default weights.
| Setting | Mean rule score | Change from default | 95% bootstrap interval |
|---|---|---|---|
| Default uniform weights | 0.1031 | 0 | 0 |
Double like_count weight | 0.1075 | +0.0043 | [-0.0005, 0.0096] |
Double follower_count weight | 0.0887 | -0.0144 | [-0.0172, -0.0117] |
Double reply_count weight | 0.1272 | +0.0240 | [0.0185, 0.0298] |
Double share_count weight | 0.0971 | -0.0061 | [-0.0107, -0.0017] |
Double engagement_rate weight | 0.0953 | -0.0079 | [-0.0152, 0.0009] |
Changing the weights affects the absolute rule score. Increasing the reply_count weight produces the largest change in the mean score, while increasing the follower_count weight lowers the mean score. Model-ranking sensitivity was also evaluated on these 36 matched episodes. DeepSeek-V4-Pro and GLM-5.2 consistently ranked first and second under all five single-objective weight perturbations, while GPT-5.6-Luna consistently ranked sixth. Ranking changes occurred only among the middle models with close scores: increasing the like_count or reply_count weight caused Gemini-3.1-Flash-Lite and GPT-5.6-Terra to exchange third and fourth place; increasing the share_count weight caused Claude-Sonnet-4.6 and Gemini-3.1-Flash-Lite to exchange fourth and fifth place. Increasing the follower_count or engagement_rate weight did not change the overall ranking. However, the model scores and rankings basically did not change.
Response to Q2. Does repeating evaluation rounds without additional independent scenarios affect the reliability of the ablation conclusions?
Thank you for your review. We repeat the experiments the results are reported in our response to W1.
Response to Q3. How does SOCIALSIM-BENCH differ from Sotopia and SocialBench in evaluation dimensions?
We will add a comparison table covering the interaction tasks, evaluation dimensions, and uses of simulation outcomes.
| Benchmark | Evaluation method | Rule-based environment | Strategy diagnosis | Simulation-to-report | Decision-report evaluation |
|---|---|---|---|---|---|
| SOTOPIA | LLM | ✗ | ✗ | ✗ | ✗ |
| SocialBench | LLM | ✗ | ✗ | ✗ | ✗ |
| AgentSociety | RULE + Data | ✓ | ✗ | ✗ | ✗ |
| SocioVerse | RULE + Data | ✗ | ✗ | ✗ | ✗ |
| SoMe | RULE + LLM | ✗ | ✗ | ✗ | ✗ |
| SandboxSocial | RULE + LLM | ✗ | ✗ | ✗ | ✗ |
| SocialSim | LLM | ✗ | ✗ | ✗ | ✗ |
| SimBench | RULE + Data | ✗ | ✗ | ✓ | ✗ |
| SocialVeil | LLM | ✗ | ✓ | ✗ | ✗ |
| PieArena | RULE + LLM | ✓ | ✗ | ✗ | ✗ |
| AgentSociety 2 | RULE + Data + LLM | ✓ | ✗ | ✓ | ✗ |
| SOCIALSIM-BENCH (our) | RULE + LLM | ✓ | ✓ | ✓ | ✓ |
Data denotes evaluation based on external or collected data; RULE denotes rule-based evaluation; LLM denotes LLM-based evaluation.
Response to Q4. Is there selection bias from choosing only Codex and Claude Code as baselines?
We compare three harnesses with different architectures, including Codex, Claude Code, and the harness designed in this paper. These settings cover different implementation approaches. Given that common open-source harnesses have similar basic designs, we believe the current comparison supports our experimental conclusions.
Response to Q5. Does deterministic state transition limit the evaluation of generalization to real social agents?
Regarding deterministic state transitions, it is important to clarify that the statement in the paper does not mean that agents' dialogue content or action choices are predetermined. Agents can autonomously generate dialogue and choose actions, and the environment executes state transitions based on the actual interaction. This is also common in social simulation frameworks.
Response to Q6. Recent formally published references should be added.
We will add recent formally published references that are directly relevant to the paper and update the corresponding preprint citations when formal publication versions are available.
Thank you for your rebuttal, I will keep current positive score.
Thank you for your response and suggestions. Your feedback has helped us improve our paper.
Thank you for your carefully responses.
First, this paper does not aim to design a single method, but to examine whether trajectories and outcomes from social simulation can support judgments about subsequent development, and to compare how different harnesses use simulation evidence. The results show that simulation evidence does not consistently improve report quality.
Second, the original appendix results came from a single experiment. We added three repeated experiments.
Claim Attribution Summary
The proportions were calculated by combining the total number of claims and category counts across the three repeated experiments. Trace-supported =
trace_grounded+inference_from_trace.The additional results show that Direct-Structured remains advantageous in some comparisons, but conditions using simulation results perform better in most comparisons.
Thank you for your reviews. The experiments are conduct to analysis. Skills and Refine are experimental conditions for testing the effects of common interventions in social simulation, rather than methods that must achieve consistent improvements across all models and domains in this paper. Their purpose is to analyze whether and under what circumstances interventions affect agent performance.