acceptodds
Under review as a conference paper at ICLR 2027

Agent Sociodrama: Social-Dynamics-Guided Persona Recasting for LLM Agent Teams

Abstract

Large language model (LLM) agents are increasingly deployed as teams, yet they fail in characteristically social ways: information holders stay silent, dominant speakers anchor groups on wrong answers, and norm violations go unsanctioned. We present Agent Sociodrama, a framework for modeling and validating evolving social dynamics in LLM agent teams. Under fixed personas, agents' emotional states and directed, asymmetric relationships are updated every round by a unified external LLM judge and shape their subsequent interactions. Across six LLM backbones and five social scenarios, we assess social fidelity through classical social-theory patterns and agreement with human judgments. The resulting trajectories exhibit structured social patterns and model-specific differences. We then introduce Trajectory-Guided Persona Recasting (TPR), which diagnoses group-dynamic failures from interaction transcripts and social trajectories and recasts agent personas before rerunning the task. Across four collaborative benchmarks, TPR improves performance from both heterogeneous and randomly initialized agent persona compositions, yielding up to 83.3%p of accuracy improvement over the initial casting. On representative distributed-information cooperation and mixed-motive bargaining tasks, we outperform SOTOPIA and AgentVerse in three of four comparisons, by up to 37.5%p in Pareto rate. These results show that validated social trajectories provide actionable signals for repairing LLM agent teams.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, multi-agent systems, role-playing, social dynamics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EtXZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Agent Sociodrama, a framework for modeling LLM-agent teams with fixed personas but dynamically evolving emotional states and directed interpersonal relationships. It further introduces Trajectory-Guided Persona Recasting (TPR), which uses interaction transcripts, task outcomes, and social trajectories to diagnose collaboration failures and modify persona compositions. The framework is evaluated across multiple LLM backbones and social scenarios, with additional human validation, and TPR is tested on several collaborative tasks.

Critical strengths
  1. The paper addresses an important and relatively underexplored aspect of multi-agent LLM systems. Modeling longitudinal social dynamics, rather than treating agents as static personas, is well motivated and potentially useful for understanding coordination failures.

  2. The empirical evaluation is broad. The paper evaluates multiple LLM backbones, different social settings, human agreement, and downstream collaborative tasks, providing substantially more evidence than a single-model or single-task demonstration.

Critical weaknesses
  1. The central contribution of social trajectories is not cleanly isolated. TPR jointly uses transcripts, outcomes, and social trajectories, while the main ablation also changes the interaction protocol and removes state updates. Therefore, the observed performance difference cannot be attributed specifically to the additional information contained in the social trajectories. This is consequential because the main novelty of the paper depends on these trajectories providing actionable signals beyond simpler alternatives.

  2. The evidence for social fidelity remains limited. The social states are largely produced and evaluated through a single external LLM judge, while the human validation uses a relatively small set of annotations and shows low inter-human agreement. The reported agreement is encouraging, but it does not fully establish that the continuous emotional and relational trajectories faithfully capture the underlying social dynamics.

Questions for authors

Can the authors provide a controlled ablation where the simulator and interaction protocol remain identical, but the recasting controller receives only transcripts and task outcomes, without access to the social trajectories? The result would materially affect whether I view the trajectory representation as a substantive contribution.

How robust are the social-dynamics conclusions to the choice of external judge? In particular, do the main trajectory patterns and downstream recasting decisions remain consistent when using different judges?

7 comments
AuthorsOpenReview ↗

Response to Reviewer EtXZ (1/3)

We thank the reviewer for recognizing the importance of modeling evolving social dynamics and the breadth of our evaluation. With the simulator and interaction protocol held fixed, the social trajectories contribute substantively to recasting, and our main conclusions are robust to replacing GPT-4o with independently developed LLM judges.

W1 / Q1. With the simulator and interaction protocol held fixed, the trajectories contribute substantively.

We respectfully note that the paper’s contribution is a chain that models, validates, and uses evolving social dynamics, and using the trajectories as controller input is one part of this chain (L091–101). The chain starts from the observation that LLM teams often fail through social interaction among their members, which existing approaches to improving agent teams leave largely unexamined (L049–053, L075–078). We therefore build Agent Sociodrama to model and measure these dynamics, validate their fidelity against social theory and human judgments across diverse LLMs and scenarios (Sec. 5.2), and use the resulting trajectories alongside transcripts and outcomes to diagnose team failures and recast personas, improving team performance across collaborative tasks (Sec. 4). The ablation in Appendix E examines the social-dynamics mechanism as a whole, covering its roles in both interaction and recasting.

Following the reviewer’s suggestion, we further isolate the contribution of the trajectories to recasting with the requested ablation. The simulator and interaction protocol are held fixed: agents speak in order of urgency, and S/R* is updated every round. The controller receives the transcript and task outcome, with no access to the social trajectories. All other settings match TPR. We ran 8 settings: Lost at Sea and Negotiation × two backbones (Grok 4.20, Gemini 3.1 Flash-Lite) × both initializations, with each casting evaluated over 12 episodes as in the paper. (We chose one ranking task with an expert reference and one mixed-motive negotiation.)

*S denotes each agent’s emotional state and R the directed relationship (favorability) between agents; both are updated every round by the judge.

MethodLost at Sea (L1 ↓)Negotiation (Pareto ↑)
Initial casting24.3831.3%
No S/R22.5835.4%
TPR18.3345.8%

L1 is the distance between the team ranking and the expert ranking (lower is better); for the Pareto rate, higher is better.

With the simulator and interaction protocol held fixed, access to the S/R trajectories lowers L1 from 22.58 to 18.33 on Lost at Sea and raises the Pareto rate from 35.4% to 45.8% on Negotiation. This gap isolates what the trajectories add beyond the transcript and task outcome, showing that the trajectory representation is a substantive contribution. We have added this comparison to Sec. 5 (Tab. 4), with details in Appendix F.1.

AuthorsOpenReview ↗

Response to Reviewer EtXZ (2/3)

W2. Several lines of evidence establish social fidelity.

We address the four points in turn.

(1) A single external judge. Our main conclusions are robust to replacing GPT-4o with independently developed LLM judges. We replaced the judge inside the simulation with Gemini 3.1 Flash-Lite and with Grok 4.20 (150 new episodes, details in Q2(a)). The scenario ordering is preserved, with Spearman correlations of 0.90 and 1.00 against GPT-4o.

We also re-scored the 50 human-labeled items with judges from other model families, each receiving exactly the prompt GPT-4o received. On these items, Claude Sonnet 4.6, DeepSeek V4-Flash and Gemini 3.1 Flash-Lite also all exceed the average human rater. Averaged over R and S, GPT-4o is the most human-aligned (85.7%), on par with Claude Sonnet 4.6 (85.0%) at a lower API cost.

JudgeRS
Avg. human rater69.7%67.8%
GPT-4o (used in paper)81.6%89.8%
Claude Sonnet 4.684.1%85.9%
DeepSeek V4-Flash78.6%85.9%
Gemini 3.1 Flash-Lite79.1%78.9%

(2) A small set of annotations. Each of the 50 items was labeled by 13 independent raters, giving 650 labels in total. This is well above common practice for validating LLM evaluators of social interaction: SOTOPIA (Zhou et al., ICLR 2024) validates its GPT-4 evaluator with two annotators per episode. Importantly, the sample size is consistent with the 50–100 instances recommended for the Alternative Annotator Test (Alt-Test; Calderon et al., 2025), which our judge passes.

(3) Low inter-human agreement. Low agreement is expected when people read relational and emotional change from short excerpts. SOTOPIA also reports low human agreement on relationship judgments (κ = 0.16), highlighting the inherent subjectivity of evaluating social dynamics. Even on MT-Bench, human–human agreement is 63% (Zheng et al., NeurIPS 2023). For this reason the standard test compares the judge with individual humans. Each rater is held out in turn; the majority of the remaining 12 serves as the reference, and both the held-out rater and the judge are scored against it.

Human avg.GPT-4o judgeJudge >= rater
R69.7%81.6%11 / 13
S67.8%89.8%12 / 13

Pooling both dimensions and bootstrapping over items (2,000 resamples), the judge leads the held-out rater by 16.6 percentage points, 95% Confidence Interval [3.1, 27.4]. It passes Alt-Test with a winning rate of 0.62 (threshold 0.5 at ε = 0.15), meaning it can stand in for a human annotator.

(4) Faithfully capturing social dynamics. The human study validates individual updates; to test whether the judge’s remaining errors accumulate along the continuous trajectories, we carry them through every round. We estimate these errors from the human labels and use them to correct all 22,685 logged updates (probabilistic misclassification bias analysis; Lash et al., 2009), repeating the correction 300 times with error rates redrawn to reflect their uncertainty.

Conclusion in the paperHoldsAfter correction
Order by favorability✓1.00 [0.90, 1.00]
Order by balance ratio✓0.90 [0.90, 1.00]
Order by repair rate✓0.80 [0.80, 0.90]
Fav. drops with conflict✓−1.00 [−1.00, −0.90]
Hospital warms (9 rds)✓+0.12 [+0.07, +0.17]
Tavern warms (9 rds)✓+0.09 [+0.05, +0.12]

Orderings are rank correlations with the original results; the conflict entry is the rank correlation between conflict intensity and favorability. Each entry is the median over the 300 corrections, with the range covering 95% of them in brackets.

The trajectories also reproduce the patterns that classical social psychology predicts for these scenarios (Heider, 1946; Cartwright & Harary, 1956). Favorability falls step by step from the cooperative Hospital scenario to the open-conflict Demolition scenario, and relationships warm over time in cooperative settings. The patterns hold under both independent judges and after correction, so they do not depend on a particular judge.

These results support the fidelity of the social trajectories: they align with human judgments, reproduce theoretically expected social dynamics, and remain robust across independent judges and corrections for judge bias. We have added these analyses to Appendix H.1–H.3 in our revision.

AuthorsOpenReview ↗

Response to Reviewer EtXZ (3/3)

Q2. The trajectory patterns and the recasting results hold under different judges.

(a) Main trajectory patterns under a different judge. In Agent Sociodrama, the judge updates S and R every round, and agents see the updated states in their next turn. Replacing the judge therefore changes both the measured states and the dialogue that follows. We re-ran the fidelity simulation with Gemini 3.1 Flash-Lite, and separately with Grok 4.20, as the S/R judge. That gives 150 new episodes over 5 scenarios × 3 generators. Final favorability:

ScenarioGPT-4o*GeminiGrok
Hospital0.670.510.48
Tavern0.480.360.28
Quest0.430.410.23
Negotiation0.360.060.19
Demolition0.180.050.02

*GPT-4o judge on the same three generators.

Spearman with GPT-4oGeminiGrok
Scenario ordering0.901.00
All 15 conditions0.900.98

“All 15 conditions” covers every scenario × generator pair.

Under both independent judges, the scenario ordering stays essentially the same, so our conclusions hold.

(b) Downstream recasting under a different judge. Different castings can repair the same team failure, so we assess downstream recasting by its effect on team performance. We re-ran TPR with Gemini 3.1 Flash-Lite and, separately, Grok 4.20 as the S/R judge for both the engine and the controller, in 16 settings each (4 tasks × 2 initializations × GPT-4o and Grok generators). GPT-4o remains the controller for diagnosis and persona selection; only the S/R judge is replaced.

S/R judgeImprovedUnchangedLower
Gemini 3.1 Flash-Lite1132
Grok 4.201141

Under both judges, the final casting improves over the initial one in 11 of 16 settings. These results demonstrate that the main social-dynamics patterns remain consistent across judges and that trajectory-guided recasting remains effective when the judge is replaced. We have added (a) and (b) to Appendix H.2.

We hope these results address the reviewer’s concerns. If you have any further questions, please feel free to let us know.

Reviewer_t4ecOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This papers studies the optimization of LLM-based multi-agent systems from a perspective of agent personas. The authors first propose a simulation process with social states to simulate the evolution of emotions and social relationship in multi-agent systems and confirmed its alignment with human society through agreement with humans in analytic experiments. Then the authors propose Trajectory-Guided Persona Recasting (TPR), which analyzes previous failure and recasts agent personas. Experiments across different backbone models and interaction settings confirm that TPR does help in achieving a better task utility.

Critical strengths
  1. This papers studies the optimization of multi-agent systems from a perspective of agent personas, which is timely and valuable.
  2. Initial analyses confirm some alignment between LLMs and humans in the evolution of emotions and relationship.
  3. Experiments shows that TPR achieves consistent advantage across different models and tasks and outperforms SOTOPIA and AgentVerse, validating its effectiveness.
Critical weaknesses
  1. The methodology seems closely related to automatic prompt engineering. The process also looks similar to dianosis framework over agents like [1, 2]. These are not well covered in the discussion.
  2. The experiments are based on relatively simple environments. I would be interested in seeing the consistence and evolution of teams in more realistic agentic tasks like search and software engineering.
  3. The presentation is not clear enough. Some processes and conclusions are not well explained and justified (see comments below).

[1] Zhu et al. Where LLM Agents Fail and How They can Learn From Failures. arXiv preprint arXiv:2509.25370.

[2] Wang et al. The long-horizon task mirage? diagnosing where and why agentic systems break. arXiv preprint arXiv:2604.11978.

Questions for authors
  1. The design of per-round process in Section 3.3 is not well justified and convincing. Specifically:
  • Action Generation: In human society, speak order does influence the results of many events, and researchers also find relevant phenomenons on LLMs [3]. Simply skipping this effect is not appropriate. I wonder whether results (both social dynamics alignment and TPR's advantage) would change under different speak orders.
  • USU_S and URU_R: Is there any reference in sociology that motivates your design? For example, why do you need to clip the delta and how are the thresholds selected for URU_R in Appendix B? Are the results sensitive to these designs?
  1. L377: This sentence only describes what STS^T refers to but does not tell what the values and trends of STS^T in Table 1 indicate.

Final affective valence STS^T reflects both initial emotional state and subsequent interactions.

  1. L380: Why does this happen and what is the insight behind this?

Despite declining favorability, this suggests that more conflictual settings can exhibit stronger bilateral alignment.

  1. L389: What does this sentence mean? Specifically, what are "ceiling and floor effects" here and what is their relationship to the outcomes?

where outcomes are less affected by ceiling or floor effects.

  1. Table 2: Can the authors explain why the init results change so significantly between w/ and w/o social dynamics?
  2. Case study: It would be better if there is a complete input-reasoning-output example of recasting. I am curious about how the model decides the change from one persona to another.
  3. L473: What are the actionable signals?

The social trajectories validated in Sec. 5.2 provide actionable signals for identifying stronger team compositions for each setting.

[3] Guan et al. MOC: Multi-Order Communication in LLM-based Multi-Agent Systems. arXiv preprint arXiv:2606.02359.

9 comments
AuthorsOpenReview ↗

Response to Reviewer t4ec (1/5)

We thank the reviewer for the constructive comments. We ran all the suggested experiments and the results further support our claims.

W1. On the Relation to Automatic Prompt Engineering and Agent Diagnosis Frameworks [1, 2]

Thank you for highlighting these connections. While TPR shares the feedback-driven refinement structure of prompt optimization and agent diagnosis, it addresses a broader team-level challenge: how evolving social dynamics cause collective failures, and how these failures can be repaired by changing team composition. The proposed TPR uses emotional and relational trajectories to diagnose breakdowns between members and guide targeted persona recasting.

Automatic prompt engineering methods, such as APE (Zhou et al., ICLR 2023) and OPRO (Yang et al., ICLR 2024), refine task instructions, and AgentDebug [1] and HORIZON [2] diagnose task-level errors such as faulty planning or tool use. However, many team failures arise between members: key information is dismissed, one member dominates, or cooperation breaks down. Such social failures depend on who plays which role. TPR therefore uses interaction transcripts and emotional and relational trajectories to diagnose these breakdowns, then repairs team composition by recasting members’ personas while preserving task instructions, task roles, and private information.

Following the reviewer’s suggestion, we have additionally provided a comparison of [1, 2] to our recasting pipeline: each framework’s diagnosis replaces the social trajectories as the controller’s input, with the engine, controller, and recasting budget unchanged (NASA Moon Survival and Lost at Sea, 8 settings each).

MethodNASA Moon Survival (L1 ↓)Lost at Sea (L1 ↓)Settings worse than the initial casting
Initial casting19.7924.38–
AgentDebug [1]16.3419.001 / 8
HORIZON [2]15.0026.125 / 8
TPR (Ours)14.1218.330 / 8

TPR achieves the lowest error on both tasks and is the only method that never leaves a team worse than it started. Under the same engine, controller, and budget, social trajectories guide more effective persona recasting than either diagnosis framework. We have added this discussion to Sec. 2 and this comparison to Appendix F.3.

AuthorsOpenReview ↗

Response to Reviewer t4ec (2/5)

W2. On the Choice of Environments and More Realistic Agentic Tasks

We would like to clarify that our tasks are standard benchmarks for LLM teams, grounded in established research paradigms and far from easy. Student Council, NASA Moon Survival, and Lost at Sea follow the multi-agent benchmarks of Pappu et al. (ICML 2026), built on classic group-decision paradigms (Stasser & Titus, 1985; Hall & Watson, 1970; Hill, 1982), and Negotiation follows the multi-issue paradigm (Raiffa, 1982) also used to evaluate LLM agents (Abdelnabi et al., NeurIPS 2024). On such tasks, LLM teams systematically fail to use distributed knowledge (Li et al., ICML 2026) and hold back their experts (Pappu et al., ICML 2026). The tasks capture coordination challenges shared across many collaborative settings, integrating distributed knowledge and reconciling conflicting interests, and directly test the social dynamics that TPR targets. Their rule-based outcome metrics also let us evaluate persona changes independently of the social-state judge. Our work studies how these social dynamics cause team failures and whether modeling them can repair the team, a question that existing agent-team methods leave largely unexamined (L075–078).

Following the reviewer’s suggestion, we further broaden the evaluation to multi-hop question answering and code review using adapted HotpotQA and HumanEval tasks. We use the original questions and passages from HotpotQA’s distractor setting and the original problems and unit tests from HumanEval, and distribute the information across members following the hidden-profile construction (Li et al., ICML 2026):

Multi-hop QA (HotpotQA). The two passages needed for the answer are held by different members, and two others hold distractor passages supporting plausible wrong answers; the team selects the answer from four candidates.

Code review (HumanEval). The function’s requirements are split across members; the team selects the correct implementation from four candidates, each incorrect one violating a different requirement.

We keep only instances that require pooling members’ information (12 per task) and ran TPR with Gemini 3.1 Flash-Lite under both initializations.

TaskHetero (Init. → Final)Random (Init. → Final)Mean (Init. → Final)
Multi-hop QA (HotpotQA)83.3% → 100%91.7% → 91.7%87.5% → 95.8%
Code review (HumanEval)83.3% → 91.7%91.7% → 100%87.5% → 95.8%

On both adapted tasks, mean accuracy across the two initializations increases from 87.5% to 95.8%. TPR reaches 100% in two of the four runs, and no final run falls below its initial accuracy. These results broaden our evaluation to collaborative multi-hop QA and code review and show that TPR can further improve teams whose initial performance already exceeds the default stopping target. We have added these evaluations to the paper (Appendix F.4).

We agree that tool-using search and software-engineering agents, where members also act on external tools, are a valuable direction, and we plan to extend Agent Sociodrama to these settings in future work.

AuthorsOpenReview ↗

Response to Reviewer t4ec (3/5)

Q1. Speaking order, clipping, and thresholds in Sec. 3.3

Speaking order. We respectfully note that Sec. 3.3 (L213–240) already clearly describes how speaking order depends on agents’ social states. Each round has two phases. First, before any agent speaks in a round, every agent decides how urgently it needs to speak, based on its persona, emotional state, relationships, and the current context. Then agents speak in descending urgency, and each agent speaks at most once per round. The design is motivated by conversational self-selection (Sacks et al., 1974) and related importance-based speaker selection in multi-agent LLM discussions (Nonomura & Mori, 2025).

Fixed and random orders are weaker designs, so we used them only as stress tests of our conclusions:

Speaking orderNo preassigned speaking positionsTurns follow the social stateSame favorability ranking as urgency order
Fixed✗✗✓
Random✓✗✓
Urgency (ours)✓✓–

Under both alternative orders, the scenario ranking of favorability is identical to the ranking under urgency ordering (Spearman 1.00). Under fixed order, TPR still lowers the error on Lost at Sea from both initial castings. Agent Sociodrama is therefore robust to speaking order, and urgency ordering remains the most realistic of the three designs.

Clipping and thresholds. Sec. 3.3 (L243–244) states the purpose of clipping: it limits abrupt changes within a single round, because relationships develop step by step (Altman & Taylor, 1973). The thresholds map favorability onto the ally, rival, and enemy relations of structural balance theory (Cartwright & Harary, 1956). They only handle gradual drift. A consequential event sets the status directly (Appendix B). Only 0.06% of the 22,685 logged relationship updates reach the ±0.3 bound, and no emotional update does. We replayed all logged updates under tighter bounds and shifted thresholds:

SettingScenario orderings (favorability, balance, repair)Conflict trendWarming (Hospital / Tavern)
Original (±0.3)–−1.00+0.30 / +0.23
Clip ±0.2✓−1.00+0.30 / +0.23
Clip ±0.1✓−1.00+0.31 / +0.23
Thresholds shifted by up to 0.1 (8 variants)✓−1.00+0.30 / +0.23

✓: same scenario ranking as the original setting (Spearman 1.00). Conflict trend: rank correlation between a scenario’s conflict intensity and its final favorability. Warming: favorability gain from the first to the last round. The reported findings do not depend on these settings.

Q2. Interpretation of final affective valence S^T (L377)

S^T is the mean emotional state of the agents at the final round (Sec. 3, L158–160; Appendix D, L689). Following the notion of core affect (Russell, 2003; L110), S captures how pleasant or unpleasant each agent feels; the interaction shapes it, but it is not directed at any particular agent. Favorability R complements S: it records how each agent regards each other agent. Tab. 1(a) shows that the two vary differently across scenarios: favorability declines steadily, whereas S^T does not. Reading them together tells the controller what kind of problem a team has. Beyond the final round, the round-by-round S trajectories show whether a member’s affect improves, deteriorates, or stays stable, which TPR reads to locate where a team breaks down (Sec. 4, L301–309). The logged diagnoses on Lost at Sea show two typical uses. When affect is low, the controller reads it as disengagement, e.g., “Emotional disengagement and rival dynamics further hindered effective communication” (Gemini, Random), and recasts the members involved toward more balanced participation. When affect and trust are high but expertise is still ignored, e.g., “the group achieved high emotional engagement and trust, [but] did not fully leverage individual expert knowledge” (DeepSeek, Hetero), the controller strengthens how the expert presents evidence, as in the Q6 case.

Reviewer_B5e8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Agent Sociodrama models LLM teams using fixed personas, evolving emotions, and directed relationships. Trajectory-Guided Persona Recasting (TPR) uses interaction transcripts, social trajectories, and task outcomes to revise persona assignments between runs. Experiments across six backbones examine social fidelity and performance on four collaborative tasks. The results suggest that persona recasting can improve collaboration, although the specific contribution of social trajectories remains unclear.

Critical strengths
  1. Persona recasting is a meaningful intervention: it changes communication tendencies while preserving each agent’s task role and private information. Directed relationships also capture differences that a single group sentiment score would miss.

  2. Model-average performance improves across all four tasks under both initializations. Task outcomes are scored by rules rather than by the social-state judge, giving the performance gains an independent evaluation basis.

Critical weaknesses
  1. The experiments do not isolate the value of social trajectories for recasting diagnosis. Appendix E’s ablation also changes the engine to fixed-order public turn-taking, confounding diagnostic information with interaction behavior. The external-framework comparisons do not resolve this. A comparison using the full engine but withholding trajectories from the controller is needed. Random persona proposals under the same evaluation budget would also help establish whether targeted diagnosis improves over simpler search.

  2. The social-fidelity conclusion exceeds the validation evidence. Human agreement is assessed on a small set of consensus excerpts, and Appendix H reports substantially more positive relationship updates from the judge than from humans (about 76% versus 40%). Using the same judge does not guarantee that comparative rankings remain unaffected, especially when its updates influence later dialogue. The current evidence supports plausible simulated patterns more clearly than longitudinal social fidelity.

Questions for authors
  1. With the interaction engine unchanged, does TPR outperform a controller receiving transcripts and outcomes but no social trajectories?

  2. Does TPR outperform random persona proposals under the same evaluation budget when final castings are evaluated on fresh episodes?

  3. What evidence supports Appendix H’s claim that the judge’s directional bias preserves comparative conclusions? Do independent judges reproduce the reported ordering?

6 comments
AuthorsOpenReview ↗

Response to Reviewer B5e8 (1/3)

We thank the reviewer for the careful reading and for two well-specified experimental requests. We ran both, and the results further strengthen the paper’s central claim.

W1 / Q1. On Isolating the Value of Social Trajectories for Recasting Diagnosis

Reflecting the constructive suggestion from the reviewer, we have run the requested ablation during the rebuttal period. The engine is unchanged: agents speak in order of urgency, and *S/R is updated every round. The controller receives the transcript and task outcome, with no S or R values. All other settings match TPR. We ran 8 settings: Lost at Sea and Negotiation × two backbones (Grok 4.20, Gemini 3.1 Flash-Lite) × both initializations, with each casting evaluated over 12 episodes as in the paper. (We chose one ranking task with an expert reference and one mixed-motive negotiation; the random-recasting comparison in Q2 uses the same two tasks.)

*S denotes each agent’s emotional state and R the directed relationship (favorability) between agents; both are updated every round by the judge.

MethodRecasting controllerS/R given to controllerLost at Sea (L1 ↓)Negotiation (Pareto ↑)
Initial casting––24.3831.3%
Controller without S/R✓✗22.5835.4%
TPR (Ours)✓✓18.3345.8%

S/R supports targeted recasting. S/R trajectories organize social feedback into evolving emotional states and directed relationships, making the direction, magnitude, and persistence of changes explicit across rounds. This structure helps the controller identify sustained interaction problems and target persona changes. With the engine unchanged, adding S/R trajectories lowers mean L1 by a further 4.25 on Lost at Sea and raises the Pareto rate by a further 10.4 percentage points on Negotiation. These gains directly support the value of social trajectories as controller input. We have added this comparison to Sec. 5 (Tab. 4), with details in Appendix F.1.

AuthorsOpenReview ↗

Response to Reviewer B5e8 (2/3)

W2. On Longitudinal Social Fidelity and Judge Reliability

The validation evidence covers each step from local updates to nine-round trajectories. We address the three concerns in turn.

(1) Human agreement on a small set of consensus excerpts. Our human validation does not rest on the consensus excerpts. Each of the 50 items (25 for R, 25 for S) was labeled by 13 independent raters, giving 650 labels in total. This is well above common practice for validating LLM evaluators of social interaction: SOTOPIA (Zhou et al., ICLR 2024) validates its GPT-4 evaluator with two annotators per episode. Importantly, the sample size is consistent with the 50–100 instances recommended for the alternative annotator test (Calderon et al., 2025).

Following Zheng et al. (2023) and Gilardi et al. (2023), we compare the judge with each individual rater: each rater is held out in turn, and both the held-out rater and the judge are scored against the majority of the remaining 12. The judge is more accurate than the average human rater on both dimensions.

Human avg.GPT-4o judgeJudge >= rater
R69.7%81.6%11 / 13
S67.8%89.8%12 / 13

Pooling both dimensions and bootstrapping over items (2,000 resamples), the judge leads the held-out rater by 16.6 percentage points, 95% CI [3.1, 27.4]. It also passes the alternative annotator test (Alt-Test; Calderon et al., 2025) on all 50 items, with a winning rate of 0.62 against a threshold of 0.5 at ε = 0.15. The judge’s reliability therefore does not rest on a small set of consensus excerpts.

(2) The judge’s positive bias. This bias does not overturn the comparative or longitudinal findings. We estimate the judge’s labeling errors from the human labels and use them to correct all 22,685 logged updates (probabilistic misclassification bias analysis; Lash et al., 2009), repeating the correction 300 times with error rates redrawn to reflect their uncertainty.

Conclusion in the paperHoldsAfter correction
Order by favorabilityYes1.00 [0.90, 1.00]
Order by balance ratioYes0.90 [0.90, 1.00]
Order by repair rateYes0.80 [0.80, 0.90]
Fav. drops with conflictYes-1.00 [-1.00, -0.90]
Hospital warms (9 rds)Yes+0.12 [+0.07, +0.17]
Tavern warms (9 rds)Yes+0.09 [+0.05, +0.12]

Orderings are rank correlations with the original results; the conflict entry is the rank correlation between conflict intensity and favorability. Each entry is the median over the 300 corrections, with the range covering 95% of them in brackets. The rankings stay strongly aligned with the original results, and both nine-round warming trends stay positive across their ranges. The judge’s positive bias therefore does not alter the paper’s longitudinal conclusions.

(3) Comparative rankings when the same judge’s updates influence later dialogue. The rankings do not hinge on GPT-4o or on its feedback into the dialogue. We replace GPT-4o in two separate tests:

TestJudges replacing GPT-4oResult
Inside the simulation: the new judge updates S/R and drives the subsequent dialogue (details in Q3)Gemini 3.1 Flash-Lite, Grok 4.20Scenario ordering preserved (Spearman 0.90 and 1.00 against GPT-4o)
On the 50 human-labeled items: each judge receives exactly the prompt GPT-4o receivedClaude Sonnet 4.6, DeepSeek V4-Flash, Gemini 3.1 Flash-LiteAll exceed the average human rater (below)
JudgeRS
Avg. human rater69.7%67.8%
GPT-4o (used in paper)81.6%89.8%
Claude Sonnet 4.684.1%85.9%
DeepSeek V4-Flash78.6%85.9%
Gemini 3.1 Flash-Lite79.1%78.9%

Averaged over R and S, GPT-4o is the most human-aligned (85.7%), on par with Claude Sonnet 4.6 (85.0%) at a lower API cost. The comparative conclusions therefore do not depend on a shared judge or on its feedback into later dialogue.

Longitudinal fidelity. Our fidelity claims concern relationship dynamics over nine rounds (Table 1, Fig. 4). Each trajectory accumulates per-round updates: the human study validates these updates, the bias correction carries their errors through all nine rounds, and the in-loop judge replacement tests their feedback into later dialogue. Together, these results support longitudinal social fidelity, beyond plausible simulated patterns. We have added the bias-correction analysis to Appendix H.1.

AuthorsOpenReview ↗

Response to Reviewer B5e8 (3/3)

Q2. On Comparison with Random Recasting under a Matched Budget

Following the reviewer’s question, we’ve conducted the comparison of ours with random persona recasting under a matched evaluation budget. In short, TPR achieves better mean performance on both tasks even though random recasting keeps its best casting out of four while TPR keeps its last. Its advantage comes from using diagnosed interaction failures to guide both whom to recast and which persona to assign, directing the search toward changes that address the team’s specific difficulties.

At each iteration, random recasting changes as many roles as TPR did and draws the new personas uniformly from the persona library. Random recasting then keeps its best casting across the four iterations, while TPR keeps its last, as in the paper. This choice favors random recasting. Both final castings are re-run on 12 fresh episodes. The tasks, backbones and initializations are the same as in Q1. We have added this comparison to Appendix F.2.

Fresh episodesLost at Sea (L1 ↓)Negotiation (Pareto ↑)
Random (best of 4)23.3729.2%
TPR (last iteration)21.0439.6%

Q3. On the Robustness of Comparative Conclusions to the Choice of Judge

The human-calibrated bias correction in W2 (2) is the direct evidence for Appendix H’s claim: correcting the judge’s directional bias preserves the comparative rankings and trends. Independent judges also reproduce the reported ordering, both inside the simulation and in TPR.

(a) Replacing the judge inside the simulation. This addresses the concern that the judge's updates feed into later dialogue. We re-ran the fidelity simulation with Gemini 3.1 Flash-Lite, and separately with Grok 4.20, as the in-loop S/R judge. That gives 150 new episodes over 5 scenarios × 3 generators. Final favorability:

ScenarioGPT-4o*GeminiGrok
Hospital0.670.510.48
Tavern0.480.360.28
Quest0.430.410.23
Negotiation0.360.060.19
Demolition0.180.050.02

*GPT-4o judge on the same three generators.

Spearman with GPT-4oGeminiGrok
Scenario ordering0.901.00
All 15 conditions0.900.98

"All 15 conditions" covers every scenario × generator pair.

Under both independent judges, the scenario ordering stays essentially the same, so our conclusions hold.

(b) Replacing the judge in TPR itself. We re-ran TPR with Gemini 3.1 Flash-Lite and, separately, Grok 4.20 as the S/R judge for both the engine and the controller, in 16 settings each (4 tasks × 2 initializations × GPT-4o and Grok generators). GPT-4o remains the controller for diagnosis and persona selection. Under both judges, the final casting improves over the initial one in 11 of 16 settings, and recasting keeps working when the trajectories come from a different judge.

S/R judgeImprovedUnchangedLower
Gemini 3.1 Flash-Lite1132
Grok 4.201141

We have added (a) and (b) to Appendix H.2.

These results show that the main social-dynamics patterns remain consistent across judges and that trajectory-guided recasting remains effective when the S/R judge is replaced.

We hope our responses address the reviewer’s concerns. If you have any further questions, please feel free to let us know.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.