- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Agent Sociodrama, a framework for modeling LLM-agent teams with fixed personas but dynamically evolving emotional states and directed interpersonal relationships. It further introduces Trajectory-Guided Persona Recasting (TPR), which uses interaction transcripts, task outcomes, and social trajectories to diagnose collaboration failures and modify persona compositions. The framework is evaluated across multiple LLM backbones and social scenarios, with additional human validation, and TPR is tested on several collaborative tasks.
Critical strengths
-
The paper addresses an important and relatively underexplored aspect of multi-agent LLM systems. Modeling longitudinal social dynamics, rather than treating agents as static personas, is well motivated and potentially useful for understanding coordination failures.
-
The empirical evaluation is broad. The paper evaluates multiple LLM backbones, different social settings, human agreement, and downstream collaborative tasks, providing substantially more evidence than a single-model or single-task demonstration.
Critical weaknesses
-
The central contribution of social trajectories is not cleanly isolated. TPR jointly uses transcripts, outcomes, and social trajectories, while the main ablation also changes the interaction protocol and removes state updates. Therefore, the observed performance difference cannot be attributed specifically to the additional information contained in the social trajectories. This is consequential because the main novelty of the paper depends on these trajectories providing actionable signals beyond simpler alternatives.
-
The evidence for social fidelity remains limited. The social states are largely produced and evaluated through a single external LLM judge, while the human validation uses a relatively small set of annotations and shows low inter-human agreement. The reported agreement is encouraging, but it does not fully establish that the continuous emotional and relational trajectories faithfully capture the underlying social dynamics.
Questions for authors
Can the authors provide a controlled ablation where the simulator and interaction protocol remain identical, but the recasting controller receives only transcripts and task outcomes, without access to the social trajectories? The result would materially affect whether I view the trajectory representation as a substantive contribution.
How robust are the social-dynamics conclusions to the choice of external judge? In particular, do the main trajectory patterns and downstream recasting decisions remain consistent when using different judges?
7 comments
Response to Reviewer EtXZ (2/3)
W2. Several lines of evidence establish social fidelity.
We address the four points in turn.
(1) A single external judge. Our main conclusions are robust to replacing GPT-4o with independently developed LLM judges. We replaced the judge inside the simulation with Gemini 3.1 Flash-Lite and with Grok 4.20 (150 new episodes, details in Q2(a)). The scenario ordering is preserved, with Spearman correlations of 0.90 and 1.00 against GPT-4o.
We also re-scored the 50 human-labeled items with judges from other model families, each receiving exactly the prompt GPT-4o received. On these items, Claude Sonnet 4.6, DeepSeek V4-Flash and Gemini 3.1 Flash-Lite also all exceed the average human rater. Averaged over R and S, GPT-4o is the most human-aligned (85.7%), on par with Claude Sonnet 4.6 (85.0%) at a lower API cost.
| Judge | R | S |
|---|---|---|
| Avg. human rater | 69.7% | 67.8% |
| GPT-4o (used in paper) | 81.6% | 89.8% |
| Claude Sonnet 4.6 | 84.1% | 85.9% |
| DeepSeek V4-Flash | 78.6% | 85.9% |
| Gemini 3.1 Flash-Lite | 79.1% | 78.9% |
(2) A small set of annotations. Each of the 50 items was labeled by 13 independent raters, giving 650 labels in total. This is well above common practice for validating LLM evaluators of social interaction: SOTOPIA (Zhou et al., ICLR 2024) validates its GPT-4 evaluator with two annotators per episode. Importantly, the sample size is consistent with the 50–100 instances recommended for the Alternative Annotator Test (Alt-Test; Calderon et al., 2025), which our judge passes.
(3) Low inter-human agreement. Low agreement is expected when people read relational and emotional change from short excerpts. SOTOPIA also reports low human agreement on relationship judgments (κ = 0.16), highlighting the inherent subjectivity of evaluating social dynamics. Even on MT-Bench, human–human agreement is 63% (Zheng et al., NeurIPS 2023). For this reason the standard test compares the judge with individual humans. Each rater is held out in turn; the majority of the remaining 12 serves as the reference, and both the held-out rater and the judge are scored against it.
| Human avg. | GPT-4o judge | Judge >= rater | |
|---|---|---|---|
| R | 69.7% | 81.6% | 11 / 13 |
| S | 67.8% | 89.8% | 12 / 13 |
Pooling both dimensions and bootstrapping over items (2,000 resamples), the judge leads the held-out rater by 16.6 percentage points, 95% Confidence Interval [3.1, 27.4]. It passes Alt-Test with a winning rate of 0.62 (threshold 0.5 at ε = 0.15), meaning it can stand in for a human annotator.
(4) Faithfully capturing social dynamics. The human study validates individual updates; to test whether the judge’s remaining errors accumulate along the continuous trajectories, we carry them through every round. We estimate these errors from the human labels and use them to correct all 22,685 logged updates (probabilistic misclassification bias analysis; Lash et al., 2009), repeating the correction 300 times with error rates redrawn to reflect their uncertainty.
| Conclusion in the paper | Holds | After correction |
|---|---|---|
| Order by favorability | ✓ | 1.00 [0.90, 1.00] |
| Order by balance ratio | ✓ | 0.90 [0.90, 1.00] |
| Order by repair rate | ✓ | 0.80 [0.80, 0.90] |
| Fav. drops with conflict | ✓ | −1.00 [−1.00, −0.90] |
| Hospital warms (9 rds) | ✓ | +0.12 [+0.07, +0.17] |
| Tavern warms (9 rds) | ✓ | +0.09 [+0.05, +0.12] |
Orderings are rank correlations with the original results; the conflict entry is the rank correlation between conflict intensity and favorability. Each entry is the median over the 300 corrections, with the range covering 95% of them in brackets.
The trajectories also reproduce the patterns that classical social psychology predicts for these scenarios (Heider, 1946; Cartwright & Harary, 1956). Favorability falls step by step from the cooperative Hospital scenario to the open-conflict Demolition scenario, and relationships warm over time in cooperative settings. The patterns hold under both independent judges and after correction, so they do not depend on a particular judge.
These results support the fidelity of the social trajectories: they align with human judgments, reproduce theoretically expected social dynamics, and remain robust across independent judges and corrections for judge bias. We have added these analyses to Appendix H.1–H.3 in our revision.
Response to Reviewer EtXZ (3/3)
Q2. The trajectory patterns and the recasting results hold under different judges.
(a) Main trajectory patterns under a different judge. In Agent Sociodrama, the judge updates S and R every round, and agents see the updated states in their next turn. Replacing the judge therefore changes both the measured states and the dialogue that follows. We re-ran the fidelity simulation with Gemini 3.1 Flash-Lite, and separately with Grok 4.20, as the S/R judge. That gives 150 new episodes over 5 scenarios × 3 generators. Final favorability:
| Scenario | GPT-4o* | Gemini | Grok |
|---|---|---|---|
| Hospital | 0.67 | 0.51 | 0.48 |
| Tavern | 0.48 | 0.36 | 0.28 |
| Quest | 0.43 | 0.41 | 0.23 |
| Negotiation | 0.36 | 0.06 | 0.19 |
| Demolition | 0.18 | 0.05 | 0.02 |
*GPT-4o judge on the same three generators.
| Spearman with GPT-4o | Gemini | Grok |
|---|---|---|
| Scenario ordering | 0.90 | 1.00 |
| All 15 conditions | 0.90 | 0.98 |
“All 15 conditions” covers every scenario × generator pair.
Under both independent judges, the scenario ordering stays essentially the same, so our conclusions hold.
(b) Downstream recasting under a different judge. Different castings can repair the same team failure, so we assess downstream recasting by its effect on team performance. We re-ran TPR with Gemini 3.1 Flash-Lite and, separately, Grok 4.20 as the S/R judge for both the engine and the controller, in 16 settings each (4 tasks × 2 initializations × GPT-4o and Grok generators). GPT-4o remains the controller for diagnosis and persona selection; only the S/R judge is replaced.
| S/R judge | Improved | Unchanged | Lower |
|---|---|---|---|
| Gemini 3.1 Flash-Lite | 11 | 3 | 2 |
| Grok 4.20 | 11 | 4 | 1 |
Under both judges, the final casting improves over the initial one in 11 of 16 settings. These results demonstrate that the main social-dynamics patterns remain consistent across judges and that trajectory-guided recasting remains effective when the judge is replaced. We have added (a) and (b) to Appendix H.2.
We hope these results address the reviewer’s concerns. If you have any further questions, please feel free to let us know.
Response to Reviewer EtXZ (1/3)
We thank the reviewer for recognizing the importance of modeling evolving social dynamics and the breadth of our evaluation. With the simulator and interaction protocol held fixed, the social trajectories contribute substantively to recasting, and our main conclusions are robust to replacing GPT-4o with independently developed LLM judges.
W1 / Q1. With the simulator and interaction protocol held fixed, the trajectories contribute substantively.
We respectfully note that the paper’s contribution is a chain that models, validates, and uses evolving social dynamics, and using the trajectories as controller input is one part of this chain (L091–101). The chain starts from the observation that LLM teams often fail through social interaction among their members, which existing approaches to improving agent teams leave largely unexamined (L049–053, L075–078). We therefore build Agent Sociodrama to model and measure these dynamics, validate their fidelity against social theory and human judgments across diverse LLMs and scenarios (Sec. 5.2), and use the resulting trajectories alongside transcripts and outcomes to diagnose team failures and recast personas, improving team performance across collaborative tasks (Sec. 4). The ablation in Appendix E examines the social-dynamics mechanism as a whole, covering its roles in both interaction and recasting.
Following the reviewer’s suggestion, we further isolate the contribution of the trajectories to recasting with the requested ablation. The simulator and interaction protocol are held fixed: agents speak in order of urgency, and S/R* is updated every round. The controller receives the transcript and task outcome, with no access to the social trajectories. All other settings match TPR. We ran 8 settings: Lost at Sea and Negotiation × two backbones (Grok 4.20, Gemini 3.1 Flash-Lite) × both initializations, with each casting evaluated over 12 episodes as in the paper. (We chose one ranking task with an expert reference and one mixed-motive negotiation.)
*S denotes each agent’s emotional state and R the directed relationship (favorability) between agents; both are updated every round by the judge.
L1 is the distance between the team ranking and the expert ranking (lower is better); for the Pareto rate, higher is better.
With the simulator and interaction protocol held fixed, access to the S/R trajectories lowers L1 from 22.58 to 18.33 on Lost at Sea and raises the Pareto rate from 35.4% to 45.8% on Negotiation. This gap isolates what the trajectories add beyond the transcript and task outcome, showing that the trajectory representation is a substantive contribution. We have added this comparison to Sec. 5 (Tab. 4), with details in Appendix F.1.