acceptodds
Under review as a conference paper at ICLR 2027

From Simulation to Evidence-Grounded Decision Analysis: A Benchmark and Decision-Support Framework for Multi-Agent Social Systems

Abstract

Most LLM-based social-simulation studies evaluate realism, social intelligence, or task success within a simulated world. We ask two related questions: whether agents exhibit adaptive social strategies beyond role-consistent responses, and how execution traces and simulation results can support disciplined decision analysis. We present , an executable benchmark with social-media and commercial-negotiation environments such as role profiles, private goals, visibility constraints, and deterministic state transitions. We evaluate agents with rule-based outcomes and five LLM-as-a-Judge dimensions, while also analyzing the strategies they employ, finding that even the excellent LLMs only perform simple strategies without applying complex methods. We further study a simulation-to-decision workflow that transforms traces into conditional reports with triggers, alternatives, trade-offs, and exit conditions. Our report ablations show that trajectory summaries are more useful than equally budgeted raw traces. Matched simulation evidence can provide additional decision-relevant content, but current results do not establish that simulation-grounded reports consistently outperform carefully structured scenario-only reports. Accordingly, evaluates trace-grounded report properties rather than real-world decision outcomes.

Primary area:
datasets and benchmarks
Keywords:
Social simulation, Decision support

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_x31YOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces SOCIALSIM BENCH, an executable multi agent benchmark covering social media and commercial negotiation scenarios. It adopts rule based metrics and five dimensional LLM as a Judge to evaluate agents’ adaptive social strategies. It further builds a simulation to decision pipeline generating conditional decision reports. Experiments reveal strong LLMs mostly use simple local strategies, and trajectory summarization improves report quality.

Critical strengths
  1. It designs a controlled executable benchmark with deterministic state transitions for multi agent social simulation, combining objective rule based outcomes and LLM as a Judge to measure both task results and interactive behavioral quality.
  2. It investigates a simulation to decision workflow, studying how simulation traces can support conditional decision analysis.
Critical weaknesses
  1. simulation grounded reports do not consistently outperform well structured scenario only reports, limiting practical decision support value.
  2. Skill and refinement interventions cannot stably improve agent performance across models and domains, lacking reliable generalizable enhancement for social agents.
Questions for authors
  1. 1In Section 3.3, the social media rule score in this paper adopts priority-weighted average. What is the basis for selecting the weight wkw_k? Why is weight ablation validation not carried out?
  2. The experiment only repeats evaluation rounds and does not use independent scenarios for ablation. Will this cause overfitting of evaluation results and reduce the reliability of the conclusion?
  3. This paper does not conduct direct comparison experiments with published social simulation benchmarks in the same field such as Sotopia and SocialBench. How does SOCIALSIM-BENCH reflect its unique technical advantages in evaluation dimensions compared with existing benchmarks?
  4. "access to raw simulation artifacts alone is insufficient", this conclusion comes from two harness baselines: Codex and Claude Code. Does selecting only these two baselines have baseline selection bias?
  5. The environment adopts fully deterministic state transition zt+1z_{t+1}. Will a fully deterministic simulation environment reduce the evaluation validity of the generalization ability of real LLM social agents?
  6. In the reference section, a considerable portion of the literature are only preprints. It is suggested to select the latest published literature from authoritative journals in the recent three years as much as possible.
4 comments

Thank you for your carefully responses.

Response to W1. Simulation reports do not consistently outperform structured reports based only on scenario information, limiting their practical value.

First, this paper does not aim to design a single method, but to examine whether trajectories and outcomes from social simulation can support judgments about subsequent development, and to compare how different harnesses use simulation evidence. The results show that simulation evidence does not consistently improve report quality.

Second, the original appendix results came from a single experiment. We added three repeated experiments.

DomainComparisonWinsTiesJudgmentsMean score (left vs. right)Variance (left vs. right)
Social mediaDirect-Structured vs. Full30187.84 vs. 8.120.0270 vs. 0.0115
Social mediaFull vs. Shuffled-Trace120187.32 vs. 6.860.4462 vs. 0.1629
Social mediaPer-Traj-Summary vs. Raw-Traj131188.31 vs. 8.190.0879 vs. 0.0137
NegotiationFull vs. Direct-Structured170188.37 vs. 8.060.0103 vs. 0.0018
NegotiationFull vs. Shuffled-Trace131188.22 vs. 7.860.1179 vs. 0.3673
NegotiationPer-Traj-Summary vs. Raw-Traj120188.34 vs. 8.060.0045 vs. 0.0048

Claim Attribution Summary

The proportions were calculated by combining the total number of claims and category counts across the three repeated experiments. Trace-supported = trace_grounded + inference_from_trace.

DomainConditionClaimsTrace-supportedKnowledgeUnsupportedNot verifiable
Social mediaDirect24128.6%14.1%48.5%8.7%
Social mediaDirect-Structured1,02665.2%21.9%5.8%7.0%
Social mediaRaw-Traj1,10866.0%23.9%5.2%4.9%
Social mediaPer-Traj-Summary1,21579.9%10.2%4.1%5.8%
Social mediaFull1,15582.5%11.3%2.4%3.7%
Social mediaShuffled-Trace20558.0%9.8%28.8%3.4%
NegotiationDirect41916.7%59.4%19.6%4.3%
NegotiationDirect-Structured84423.1%47.6%20.7%8.5%
NegotiationRaw-Traj1,08018.8%54.2%18.8%8.2%
NegotiationPer-Traj-Summary75152.6%38.3%2.9%6.1%
NegotiationFull1,05847.4%41.3%6.7%4.5%
NegotiationShuffled-Trace89724.1%53.8%12.9%9.1%

The additional results show that Direct-Structured remains advantageous in some comparisons, but conditions using simulation results perform better in most comparisons.

Response to W2. The benefits of Skills and Refine cannot be consistently reproduced across models and domains.

Thank you for your reviews. The experiments are conduct to analysis. Skills and Refine are experimental conditions for testing the effects of common interventions in social simulation, rather than methods that must achieve consistent improvements across all models and domains in this paper. Their purpose is to analyze whether and under what circumstances interventions affect agent performance.

Response to Q1. How were the weights wkw_k for the social-media rule score determined? Why was there no weight ablation?

Uniform weights were used by default, with each objective assigned a weight of 1. The additional sensitivity analysis is as follows. The results are based on 36 matched episodes, and the rule-objective coverage is 100%. Under the default setting, the five deterministic objectives use equal weights, with each objective assigned wk=1w_k=1 before normalization. In each sensitivity setting, only one objective's weight is doubled and the other four remain at their default weights.

SettingMean rule scoreChange from default95% bootstrap interval
Default uniform weights0.103100
Double like_count weight0.1075+0.0043[-0.0005, 0.0096]
Double follower_count weight0.0887-0.0144[-0.0172, -0.0117]
Double reply_count weight0.1272+0.0240[0.0185, 0.0298]
Double share_count weight0.0971-0.0061[-0.0107, -0.0017]
Double engagement_rate weight0.0953-0.0079[-0.0152, 0.0009]

Changing the weights affects the absolute rule score. Increasing the reply_count weight produces the largest change in the mean score, while increasing the follower_count weight lowers the mean score. Model-ranking sensitivity was also evaluated on these 36 matched episodes. DeepSeek-V4-Pro and GLM-5.2 consistently ranked first and second under all five single-objective weight perturbations, while GPT-5.6-Luna consistently ranked sixth. Ranking changes occurred only among the middle models with close scores: increasing the like_count or reply_count weight caused Gemini-3.1-Flash-Lite and GPT-5.6-Terra to exchange third and fourth place; increasing the share_count weight caused Claude-Sonnet-4.6 and Gemini-3.1-Flash-Lite to exchange fourth and fifth place. Increasing the follower_count or engagement_rate weight did not change the overall ranking. However, the model scores and rankings basically did not change.

Response to Q2. Does repeating evaluation rounds without additional independent scenarios affect the reliability of the ablation conclusions?

Thank you for your review. We repeat the experiments the results are reported in our response to W1.

Response to Q3. How does SOCIALSIM-BENCH differ from Sotopia and SocialBench in evaluation dimensions?

We will add a comparison table covering the interaction tasks, evaluation dimensions, and uses of simulation outcomes.

BenchmarkEvaluation methodRule-based environmentStrategy diagnosisSimulation-to-reportDecision-report evaluation
SOTOPIALLM✗✗✗✗
SocialBenchLLM✗✗✗✗
AgentSocietyRULE + Data✓✗✗✗
SocioVerseRULE + Data✗✗✗✗
SoMeRULE + LLM✗✗✗✗
SandboxSocialRULE + LLM✗✗✗✗
SocialSimLLM✗✗✗✗
SimBenchRULE + Data✗✗✓✗
SocialVeilLLM✗✓✗✗
PieArenaRULE + LLM✓✗✗✗
AgentSociety 2RULE + Data + LLM✓✗✓✗
SOCIALSIM-BENCH (our)RULE + LLM✓✓✓✓

Data denotes evaluation based on external or collected data; RULE denotes rule-based evaluation; LLM denotes LLM-based evaluation.

Response to Q4. Is there selection bias from choosing only Codex and Claude Code as baselines?

We compare three harnesses with different architectures, including Codex, Claude Code, and the harness designed in this paper. These settings cover different implementation approaches. Given that common open-source harnesses have similar basic designs, we believe the current comparison supports our experimental conclusions.

Response to Q5. Does deterministic state transition limit the evaluation of generalization to real social agents?

Regarding deterministic state transitions, it is important to clarify that the statement in the paper does not mean that agents' dialogue content or action choices are predetermined. Agents can autonomously generate dialogue and choose actions, and the environment executes state transitions based on the actual interaction. This is also common in social simulation frameworks.

Response to Q6. Recent formally published references should be added.

We will add recent formally published references that are directly relevant to the paper and update the corresponding preprint citations when formal publication versions are available.

Thank you for your rebuttal, I will keep current positive score.

Thank you for your response and suggestions. Your feedback has helped us improve our paper.

Reviewer_tBQqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduce a multi-agent benchmark, SOCIALSIM-BENCH, with two environments: social-media interaction and multi-party commercial negotiation. LLMs generate the scenarios, rule-based modules convert them into executable environments, and the authors manually verify the result, giving 96 negotiation and 36 social-media scenarios. Focal agents are scored with a rule-based outcome metric and five LLM-as-a-Judge dimensions. The authors also classify the strategies agents use and test two interventions and refine.

The second part of the paper is a simulation-to-decision pipeline that summarizes trajectories, aggregates recurring mechanisms, and produces conditional decision reports. It is compared against direct LLM reports and against Codex and Claude Code harnesses.

The main claimed contributions are:

  1. The benchmark itself.
  2. A cross-domain analysis showing that even strong LLMs mostly use simple strategies.
  3. A trace-grounded report harness.
Critical strengths
  1. There are explicit state tuples, legal action sets, visibility rules, and deterministic transitions make outcomes reproducible and auditable.
  2. The strategy analysis in Table 3 shows an interesting finding. It shows a clear gap between frequent local strategies and nearly absent structural ones.
  3. The report ablations are well designed. The Shuffled-Trace control and the Per-Traj-Summary vs Raw-Traj comparison isolate useful factors. The claim-level attribution analysis is a reasonable attempt to separate trace-grounded claims from general model knowledge.
Critical weaknesses
  1. The paper's goal is to measure adaptive, strategic social reasoning beyond role-consistent behavior. However, neither benchmark metric measures strategy directly. The claimed finding "even strong LLMs only use simple strategies" comes from a separate post-hoc strategy labeler (Table 3), which is not part of the benchmark's scoring. Also, the strategy usage is reported only pooled across models, and never per model. It is never linked to rule or judge scores.

  2. The LLM judge appears to still score language instead of strategy. In Appendix C.1, refine changes only the natural-language realization while preserving action types, numeric terms, and commitments. Yet it lowers the negotiation judge mean while Claude's rule score barely moves. In social media, refine raises Claude's judge mean by 3.9% while its rule score falls 24.5%. The "social reasoning" dimension correlates with the rule score at only 0.05 (Figure 2), so judged reasoning does not predict outcomes. Also the human validation is weak. The paper does not report annotator count, inter-annotator agreement, or the protocol.

  3. Both the agent scores and the report pipeline depend on the environments behaving, but the simulator itself is not validated. It is unclear whether complex strategies are rewarded by the environment. If they are not rewarded, or are hard to execute given the action space, their lack of existence reflects environment design rather than agent capability.

  4. The decision-report component evaluates the report generator with its own separate metrics. It does not use the benchmark's agent scores, so it reads as a separate project.

  5. There are already some closely related work on both negotiation ([1,2,3,4]) and social media simulation ([5]), yet the authors didn't mention them in the paper.

[1] Liu, Xianyang, Shangding Gu, and Dawn Song. "Agenticpay: A multi-agent llm negotiation system for buyer-seller transactions." arXiv preprint arXiv:2602.06008 (2026).

[2] Abdelnabi, Sahar, et al. "Cooperation, competition, and maliciousness: Llm-stakeholders interactive negotiation." Advances in Neural Information Processing Systems 37 (2024): 83548-83599.

[3] Benac, Leo, et al. "A Benchmark for Multi-Party Negotiation Games from Real Negotiation Data." arXiv preprint arXiv:2603.14066 (2026).

[4] Bianchi, Federico, et al. "How well can llms negotiate? negotiationarena platform and analysis." arXiv preprint arXiv:2402.05863 (2024).

[5] Yang, Ziyi, et al. "Oasis: Open agent social interaction simulations with one million agents." arXiv preprint arXiv:2411.11581 (2024).

Questions for authors
  1. How many episodes were run per model and scenario? Table 3 implies about 432 social-media and 1152 negotiation episodes. How are these distributed across models?
  2. Do episodes where the focal agent uses high-difficulty strategies achieve higher rule scores?
  3. Could you provide more details about human annotations?
  4. For the report evaluation, how many independent scenarios and human raters were involved? Which model served as the report judge?
5 comments

Thank you for your carefully responses.

Response to W1. Strategy annotation is independent of benchmark scoring, and per-model results and the relationship with scores are missing.

The original design used rule scores and LLM scores to measure the overall performance of social agents, while strategy annotation was used for supplementary diagnosis; we did not treat strategy as part of the evaluation. We will include per-model strategy statistics in the benchmark results in the revision and clarify their relationship with the two primary scores.

We added AboveK@. For each trajectory, we first calculate the proportion of high-difficulty strategy categories among all adopted strategy categories, and then calculate the proportion of trajectories for which this proportion is strictly greater than KK. A category is counted at most once per trajectory. Above0@ is the proportion of trajectories adopting at least one high-difficulty strategy, and Above0.50@ is the proportion in which high-difficulty categories account for more than half of the adopted categories.

DomainModelEpisodesAbove0@Above0.25@Above0.50@Above0.75@
Commercial negotiationClaude-Sonnet-4.619299.48%98.44%84.38%19.79%
Commercial negotiationDeepSeek-v4-Pro9680.21%79.17%57.29%10.42%
Commercial negotiationGemini-3.1-Flash-Lite9693.75%91.67%42.71%5.21%
Commercial negotiationGLM-5.296100.00%100.00%90.62%15.62%
Commercial negotiationGPT-5.6-Luna19292.71%92.71%61.46%5.21%
Commercial negotiationGPT-5.6-Terra9686.46%85.42%56.25%6.25%
Social mediaClaude-Sonnet-4.67218.06%12.50%2.78%0.00%
Social mediaDeepSeek-v4-Pro368.33%8.33%0.00%0.00%
Social mediaGemini-3.1-Flash-Lite3619.44%16.67%2.78%0.00%
Social mediaGLM-5.23625.00%16.67%0.00%0.00%
Social mediaGPT-5.6-Luna7216.67%16.67%2.78%0.00%
Social mediaGPT-5.6-Terra368.33%5.56%0.00%0.00%

Across models, Above0@ is 80.21%–100.00% in commercial negotiation and 8.33%–25.00% in social media. The relationship between strategy and judge score refer to Response to Q2

Response to W2. The LLM evaluator appears to evaluate language rather than strategy. In Appendix C.1, refine changes only the natural-language expression while preserving action types, numerical terms, and commitments. However, it lowers the evaluation mean in negotiation while Claude's rule score remains nearly unchanged. In social media, refine increases Claude's evaluation mean by 3.9% but decreases the rule score by 24.5%. The correlation between the "social reasoning" dimension and the rule score is only 0.05 (Figure 2), so the reasoning score provided by the evaluator cannot predict the outcome. LLM evaluation may be affected by language expression, and details of human validation are insufficient.

Rule scores cannot fully reflect a model's strategy-use capability, so we use LLM evaluation as well. Rule scores mainly measure observable outcomes such as final contracts; even if an agent adopts a more reasonable strategy, its rule score may remain low if no contract is reached. LLM evaluation can assess strategy and behavior using the complete interaction process rather than only the final outcome.

Refine changes not only language expression but may also change the subsequent interaction process. For example, a change in Agent A's wording may affect Agent B's strategy choice, which in turn changes subsequent behavior and the final outcome; therefore, the rule score may also decrease. We use the two evaluation metrics to measure different aspects of the trajectories, as explained in Section 5.4.

The additional experiment was independently completed by two raters with computer science backgrounds and one with an economics background. Each evaluated 36 negotiation episodes and 36 social-media episodes using the same five-dimensional criteria. The human reference score was first calculated as the equally weighted mean across the five dimensions and then as the equally weighted mean across the three raters; the LLM total score used the same equally weighted mean across the five dimensions.

ScopeEpisodesHuman meanLLM meanPearson rr [95% CI]pp-valueBiasMAE
Equal-weighted average across both domains2 × 366.316.870.765 [0.628, 0.851]—0.5670.593
Commercial negotiation366.406.990.774 [0.542, 0.895]3.03×10−83.03\times10^{-8}0.5930.619
Social media366.216.760.756 [0.602, 0.870]9.79×10−89.79\times10^{-8}0.5410.567

The three human raters are anonymized as A, B, and C. The following pairwise statistics are equally weighted across the two domains.

Rater pairEpisodesPearson rrMean difference (latter minus former)MAE
A / B2 × 360.2820.0280.578
A / C2 × 360.457−0.2860.797
B / C2 × 360.188−0.3140.875

Response to W3. The simulator lacks validation, and the low frequency of complex strategies may result from the action space or reward design.

The environment may affect strategy use, but our environment does not provide additional rewards for particular strategies or complex actions. Agents autonomously choose actions, and the difficulty classification of predefined strategies is used only for retrospective diagnosis and does not directly participate in reward calculation. Therefore, strategy use reflects the model's behavior in the environment rather than reward incentives targeting high-difficulty strategies.

Response to W4. The decision reports and the agent benchmark use different metrics, and their relationship is unclear.

Simulation evaluation and decision-report evaluation are two connected stages with different evaluation targets, so they cannot be measured with a single metric. Evaluating them separately helps distinguish problems in simulated behavior from problems in report generation. The simulation stage evaluates the model's capability as a social agent, while the report stage evaluates its capability to use simulation results to generate strategic recommendations. They are progressive stages; their metrics should neither be mixed nor treated as completely independent. The simulation stage evaluates the model's behavior as a social agent and the environmental outcomes. The report stage uses the trajectories and outcomes produced in the preceding stage to evaluate whether the model can transform these materials into decision recommendations. The connection between the two stages is the transfer of evidence, while each set of metrics evaluates its corresponding output.

Response to W5. Related work omits negotiation studies [1–4] and social-media simulation studies [5].

Thank you for your suggestions. We will add these references to the related-work section.

Response to Q1. How many episodes were run for each model and scenario? How is the total in Table 3 distributed?

Each episode corresponds to one simulation run and one trajectory, and the same trajectory may receive multiple strategy labels. The number of episodes used for the per-model analysis is shown in the AboveK@ table: 768 for commercial negotiation and 288 for social media. Each episode corresponds to one test instance and its simulation trajectory; different models use the same test data. For example, the negotiation scenario in Table 3 contains 96 test instances, and each model has 96 corresponding episodes on that scenario. Because one trajectory can be annotated with multiple strategy categories, the number of strategy occurrences is not equal to the number of episodes.

Response to Q2. Do episodes that use high-difficulty strategies obtain higher rule scores?

Under the Basic condition, we grouped episodes according to whether the target agent used at least one predefined high-difficulty strategy, excluded other_strategy, and counted each episode in only one group. Rule scores range from 0 to 1.

DomainAdopted group, nAdopted group meanNot-adopted group, nNot-adopted group meanMean difference
Commercial negotiation7150.241530.160+0.081
Social media470.1772410.111+0.066

The adopted group has a higher rule-score mean in both domains.

Response to Q3. Can details of the human annotation be added?

The annotator backgrounds, procedure, and inter-rater consistency results for the human evaluation are provided in the response to W2 for this reviewer.

Response to Q4. How many independent scenarios and human raters were involved in the report evaluation? Which model served as the report evaluator?

The LLM evaluator for the reports was GPT-5.6-Terra. Each scenario used 32 data items to generate a report, human report evaluation was conducted by two graduate student.

Reviewer_ubDSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SocialSim-Bench, an executable benchmark comprising 96 commercial-negotiation and 36 social-media scenarios. Its environments specify roles, private goals, information visibility, action constraints, and deterministic state transitions. Focal agents are evaluated using rule-based outcomes and five LLM-as-a-Judge dimensions. The authors compare six models, analyze strategy usage and skills/refine interventions, and propose a simulation-to-decision pipeline that summarizes individual trajectories, aggregates patterns, and produces conditional recommendations. Results show domain- and metric-dependent model rankings, inconsistent intervention effects, and advantages of trajectory summaries over raw traces. The incremental benefit of simulation evidence over structured scenario-only reporting remains domain dependent.

Critical strengths
  1. Executable interaction with auditable outcomes. Explicit private goals, visibility constraints, action restrictions, and contract states allow evaluation to extend beyond conversational plausibility and persona consistency. Keeping other participants on a fixed model while varying the focal agent also provides a reasonable basis for controlled comparison.

  2. Meaningful report-generation controls and transparent limitations. Direct-Structured controls for report structure, Shuffled-Trace examines evidence relevance, and Raw-Traj versus Per-Traj-Summary examines evidence organization. The abstract and Section 5.7 appropriately acknowledge that simulation evidence does not consistently outperform structured scenario-only reports, yielding a useful empirical finding with a clearly bounded scope.

Critical weaknesses
  1. The validity of benchmark scoring and strategic diagnoses is not sufficiently established. Equation 4 leaves the activity component undefined, while Appendix A states that weights and thresholds are fixed without providing their numerical values. It is also unclear how a mixture of buyer savings, seller sustainability, and activity reflects the focal agent’s particular role and private goals. These omissions affect interpretation of both scores and model rankings. Human validation of the LLM judge covers only 18 episodes per domain; negotiation has r=0.37r=0.37 and p=0.13p=0.13, without adequate details on annotator numbers, backgrounds, or agreement. Although strategy categories are defined, the annotation protocol and opportunities for using each strategy are insufficiently characterized. Several strategies labeled high difficulty are common in Table 3, including competitor leverage at 79.3% and dynamic adaptation at 38.2%. The evidence therefore supports the rarity of some strategies more clearly than the broad claim that agents use only simple strategies.

  2. The independent contribution of simulation evidence to decision reports remains insufficiently supported. In Table 6, Direct-Structured defeats Full in all six social-media judgments. In negotiation, Full wins four judgments, but both methods have the same mean score of 8.32. Matched evidence outperforming mismatched evidence is informative, but does not establish an advantage over the no-evidence control. Moreover, these six judgments repeat report orders and evaluation rounds rather than independent scenarios. The numbers of independent scenarios, report instances, and human evaluators, together with uncertainty estimates, are not clearly reported for Tables 4 and 6. Consequently, Table 4’s 100% LLM win rate is difficult to interpret as generalizable evidence. The current results support organizing traces through summarization more strongly than they support an additional benefit from simulation itself.

Questions for authors
  1. Scoring clarification: Please provide the activity function in Equation 4, the definition and possible values of the contract-status factor, all numerical thresholds and weights, and aggregation across contracts and focal roles. Is the rule score intended to measure private-goal attainment, overall transaction quality, or a combination?

  2. Evaluation validation: How many people conducted human evaluation and strategy annotation, how were samples selected, were annotations blinded, and what was inter-annotator agreement? Can you provide stronger human validation for negotiation and report key strategy frequencies relative to opportunities for using them, to distinguish capability limitations from lack of opportunity?

  3. Report experiments: How many independent scenarios, reports per scenario, and human evaluators underlie Tables 4 and 6? Please provide scenario-level paired Full versus Direct-Structured results with uncertainty, under matched models, instructions, output lengths, and explicitly defined computational budgets. Please also distinguish which budgets are matched from additional calls used by the full pipeline.

These questions concern the interpretation and reliability of the current results; their answers could change my assessment.

4 comments

Thank you for your carefully responses.

Response to W1. Validity of the scoring and strategy diagnosis. Equation 4 does not define activity; the appendix does not provide the weights and thresholds, nor explain how the rule score reflects the target agent's role and private goals. The original human-validation sample was limited, and details about the annotation procedure and consistency were insufficient. Some difficult strategies are relatively common, so the conclusion that "agents only use simple strategies" is too strong.

The relevant details of the scoring definition and strategy diagnosis are as follows.

Rule scoring. Equation 4 defines activity as:

factivity=clip(nsess/2,0,1),f_{activity}=clip(nₛₑₛₛ/2,0,1),

where nsessnₛₑₛₛ is the number of completed negotiation sessions in an episode. The activity score is 0, 0.5, or 1 for 0, 1, or at least 2 sessions, respectively.

The rule score weights buyer savings, seller sustainability, and activity at 0.60, 0.15, and 0.25. The seller-margin normalization threshold is 0.05, and the contract-effectiveness threshold is 0.75. Contract-state factors are proposed = 0.25, amended = 0.50, accepted = 0.75, and signed = 1.00; rejected, failed, and unknown states receive 0. Thus, accepted and signed meet the effectiveness threshold.

The rule score captures observable transaction outcomes and quantifiable goal attainment. The LLM evaluator primarily assesses whether the target agent acts consistently with its role and pursues its private goals. These metrics capture different aspects of an episode, as discussed in Section 5.4 and Figure 2.

When an episode has multiple settleable contracts, each is scored separately before aggregation by contract type. For contract c, let rcr_{c}, pcp_{c}, kck_{c}, and vcv_{c} denote its reference price, transaction price, cost price, and volume. Let ec=1e_{c}=1 for a valid contract and 0 otherwise. Buyer savings and seller returns are Bc=ecmax(0,rc−pc)B_{c}=e_{c}max(0,r_{c}-p_{c}) and Sc=ecmax(0,pc−kc)S_{c}=e_{c}max(0,p_{c}-k_{c}); multiplying by vcv_{c} gives their totals. If multiple agents share one side, its total is divided equally: the target agent receives vcBc/nbv_{c}B_{c}/n_{b} as a buyer or vcSc/nsv_{c}S_{c}/nₛ as a seller.

For buyer-seller contracts, the score combines buyer savings ratio, seller profit margin, and activity:

sc=ec(0.60qc+0.15mc+0.25ac).s_{c}=e_{c}(0.60q_{c}+0.15m_{c}+0.25a_{c}).

Within an episode, economic returns are summed across contracts and transaction prices are volume-weighted. The final score is the highest contract score, max⁡csc\max_c s_c. For resource-allocation contracts, it is the mean of the contract scores.

Human validation. Thank you for your suggestion. The sample size used in the human evaluation was too small. To supplement the original human-evaluation sample and provide details about the annotation procedure, three raters independently evaluated 36 episodes from commercial negotiation and 36 episodes from social media, for 72 episodes in total. Two raters were computer science graduate students and one was an economics graduate student. Each rater used the same five-dimensional criteria as the benchmark. We first took the equally weighted mean of the five-dimensional scores for each rater, and then the equally weighted mean across the three raters to obtain the human reference score for each episode. The LLM total score used the same equally weighted mean across the five dimensions.

ScopeEpisodesHuman meanLLM meanPearson rr [95% CI]pp-valueMean bias (LLM minus human)MAE
Equal-weighted average across both domains2 × 366.316.870.765 [0.628, 0.851]—+0.5670.593
Commercial negotiation366.406.990.774 [0.542, 0.895]3.03×10−83.03\times10^{-8}+0.5930.619
Social media366.216.760.756 [0.602, 0.870]9.79×10−89.79\times10^{-8}+0.5410.567

We also compared the scores of the three human raters. The table below first computes pairwise statistics within each domain and then takes the equally weighted average across the two domains.

Rater pairEpisodesPearson rrMean difference (latter minus former)MAE
A / B2 × 360.282+0.0280.578
A / C2 × 360.457−0.2860.797
B / C2 × 360.188−0.3140.875

LLM scores show a strong positive correlation with the average of the three human scores in both domains. Human validation used random sampling and independent blind evaluation. All raters followed the same criteria and could not see the results of the other raters.

Response to W1. Validity of the scoring and strategy diagnosis. Equation 4 does not define activity; the appendix does not provide the weights and thresholds, nor explain how the rule score reflects the target agent's role and private goals. The original human-validation sample was limited, and details about the annotation procedure and consistency were insufficient. Some difficult strategies are relatively common, so the conclusion that "agents only use simple strategies" is too strong.

Strategy diagnosis. For the annotation protocol and opportunities for using each strategy, in our experiments, we predefined the strategy categories, difficulty levels, and evidence criteria for each domain, and then had an LLM retrospectively annotate the saved target-agent trajectories. Negotiation annotation used actions and messages; social-media annotation used action events and posts. Annotators were required to provide trajectory evidence. The occurrence rate of each strategy uses all trajectories in that domain as the denominator.

DomainTrajectoriesHigh-difficulty categoriesCategories with occurrence below 20%Category proportion
Commercial negotiation1,15211763.64%
Social media43299100.00%
Total1,584201680.00%

In both domains, many predefined difficult strategies occur in fewer than 20% of trajectories. We will narrow the original conclusion to: some predefined difficult strategies occur infrequently in the observed trajectories.

Response to W2. Independent contribution of simulation evidence to report quality. In social media, Direct-Structured beat Full in all six comparisons; in negotiation, Full won four times, although both had a mean score of 8.32. The six judgments resulted from repeating the report order and evaluation round. Tables 4 and 6 lack the number of independent samples and uncertainty information.

First, the focus of this paper is not to propose a single report-generation method, but to study whether trajectories and outcomes produced by social simulation can support judgments about subsequent development, and to compare how different harnesses use simulation evidence to generate reports.

Second, We have added more experimental results to support our conclusions. The original appendix results came from a single experiment and may be affected by randomness. We added three repeated experiments, with the following results:

DomainComparisonWinsTiesMean score (left vs. right)Variance (left vs. right)
Social mediaDirect-Structured vs. Full307.84 vs. 8.120.0270 vs. 0.0115
Social mediaFull vs. Shuffled-Trace1207.32 vs. 6.860.4462 vs. 0.1629
Social mediaPer-Traj-Summary vs. Raw-Traj1318.31 vs. 8.190.0879 vs. 0.0137
NegotiationFull vs. Direct-Structured1708.37 vs. 8.060.0103 vs. 0.0018
NegotiationFull vs. Shuffled-Trace1318.22 vs. 7.860.1179 vs. 0.3673
NegotiationPer-Traj-Summary vs. Raw-Traj1208.34 vs. 8.060.0045 vs. 0.0048

Claim Attribution Summary

The proportions were calculated by combining the total number of claims and category counts across the three repeated experiments. Trace-supported = trace_grounded + inference_from_trace.

DomainConditionClaimsTrace-supportedKnowledgeUnsupportedNot verifiable
Social mediaDirect24128.6%14.1%48.5%8.7%
Social mediaDirect-Structured1,02665.2%21.9%5.8%7.0%
Social mediaRaw-Traj1,10866.0%23.9%5.2%4.9%
Social mediaPer-Traj-Summary1,21579.9%10.2%4.1%5.8%
Social mediaFull1,15582.5%11.3%2.4%3.7%
Social mediaShuffled-Trace20558.0%9.8%28.8%3.4%
NegotiationDirect41916.7%59.4%19.6%4.3%
NegotiationDirect-Structured84423.1%47.6%20.7%8.5%
NegotiationRaw-Traj1,08018.8%54.2%18.8%8.2%
NegotiationPer-Traj-Summary75152.6%38.3%2.9%6.1%
NegotiationFull1,05847.4%41.3%6.7%4.5%
NegotiationShuffled-Trace89724.1%53.8%12.9%9.1%

The additional results show that Direct-Structured still has an advantage in some comparisons, but conditions that use simulation results perform better in most comparisons.

Response to Q1. What does the rule score measure? How are contract states, thresholds, weights, and aggregation across contracts and roles defined?

The activity, contract-state factor, weights, and thresholds are given above. The rule score measures observable transaction performance, quantifiable benefits, and goal attainment in the environmental outcomes; the LLM evaluation further examines role consistency and private goals. Both types of evaluation involve rule-based scoring. The relevant definitions and statistics are provided in the response to W1.

Response to Q2. Evaluation validation: How many people conducted human evaluation and strategy annotation, how were samples selected, were annotations blinded, and what was inter-annotator agreement? Can you provide stronger human validation for negotiation and report key strategy frequencies relative to opportunities for using them, to distinguish capability limitations from lack of opportunity?

The relevant definitions and statistics are provided in the response to W1.

Response to Q3. Report experiments: How many independent scenarios, reports per scenario, and human evaluators underlie Tables 4 and 6? Please provide scenario-level paired Full versus Direct-Structured results with uncertainty, under matched models, instructions, output lengths, and explicitly defined computational budgets. Please also distinguish which budgets are matched from additional calls used by the full pipeline.

Three reports were generated for each scenario based on 32 scenarios and evaluated by two graduate students, one in computer science and one in social science. Because the methods differ, their computational budgets are not directly comparable: the Full method requires additional calls, and its prompt also differs. The repeated report experiments and paired results are provided in the response to W2.

Dear Reviewer,

Are you having any further questions or concerns?

We would be grateful if you could review our response again and consider the work we have done to address your concerns. If you feel it is appropriate, we would appreciate your reconsidering your rating and considering a one-point increase.

Thank you sincerely for your time, understanding, and kindness. We hope that your own research efforts are met with the same care and consideration.

The authors

Comments on the paper

1 comment

Dear AC and PC,

Thank you for considering our submission and the rebuttal discussion. We summarize the paper’s main strengths and the changes we made in response to each reviewer’s concerns.

1. Strengths of the paper

  • Executable environments for social simulation. SOCIALSIM-BENCH provides social-media and commercial-negotiation environments with explicit rules, private goals, and visibility constraints. This setup supports complementary analyses of observable outcomes, role-consistent behavior, and the strategies agents use.
  • A concrete workflow from simulation to decision analysis. The simulator produces trajectories and outcomes that serve as evidence for reports specifying triggers, alternatives, trade-offs, and exit conditions.
  • Controlled comparisons of report generation. The structured scenario-only baseline, shuffled traces, and raw-versus-summarized trajectories help identify how report structure, evidence relevance, and evidence organization affect the resulting reports. These comparisons show where simulation evidence can strengthen claim grounding and where a structured scenario-only report remains competitive.

2. Reviewer concerns addressed in the rebuttal

  • Rule-score definitions and weight sensitivity (ubDS and x31Y). For ubDS, we specified how the negotiation score handles activity, weights, thresholds, and contract aggregation, and clarified its relationship to the judge’s assessment of role consistency and private-goal pursuit. For x31Y, we explained the choice of uniform weights for the social-media score and added a sensitivity analysis.
  • Strategy use across models and its relationship to outcomes (ubDS and tBQq). For tBQq, we added per-model, per-domain strategy results and examined the association between high-difficulty strategy use and rule scores. In response to ubDS, we documented the annotation procedure and narrowed the original claim to the observed frequency of specific predefined strategies.
  • Human validation and rubric reliability (ubDS and tBQq). We expanded the human evaluation and documented the independent rating procedure in response to both reviewers. We think it is enough to prove the effectiveness of our evaluation.
  • The connection between agent simulation and report evaluation (tBQq). We clarified that the report stage consumes the trajectories and outcomes generated by the benchmark.
  • The contribution of simulation evidence to report quality (ubDS, tBQq, and x31Y). We repeated the report ablations and added claim-level attribution results. The repeated comparisons favored matched over shuffled traces and summarized over raw trajectories.

We believe our rebuttal addressed all of Reviewer tBQq’s concerns, including the points raised in their final response. We received no further comments from the reviewer after our follow-up. In light of the clarifications and additional evidence we provided, we do not find the reviewer’s stated rationale for maintaining their original rating persuasive. Thank you for your time and consideration.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.