acceptodds
Under review as a conference paper at ICLR 2027

Concordia Island: A Sandbox World for Evaluating the Psychology of Generative Agents

Abstract

Generative agents have demonstrated promising fidelity in simulating human behavior on lab experiments and surveys. But can such agents sustain coherent lives and psychological responses across an entire simulated society with real-world complexity? We introduce Concordia Island, a scalable, open-ended sandbox world in which 100+ agents autonomously live, work, socialize, and adapt to significant life events. To systematically evaluate these simulations, we present a comprehensive evaluation suite for synthetic psychology. This suite integrates validated psychometric instruments for measuring emotion, personality, and life satisfaction with open-ended reflections about meaning and identity to audit the validity of agents' constructed inner experiences. We organize our evaluation around three classes of metrics: 1. longitudinal tracking of agent baselines and their responses to exogenous shocks such as job displacement, 2. psychometric reliability and validity and 3. qualitative analysis via an automated ethnographer and qualitative coding pipelines for systematically evaluating the large simulation artifacts that are produced. Across a model comparison spanning agent decision-logics and base LLMs, we demonstrate that agent self-reports exhibit reliable factor structure, context-appropriate emotional responses to life events, and meaningful individual differences—while also revealing systematic failure modes such as unrealistic attractor basins in long-run simulations.

Primary area:
datasets and benchmarks
Keywords:
LLM simulation, Social evaluation, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_p9keOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Concordia Island, a multi-agent simulation environment built on the Concordia framework, and proposes a comprehensive evaluation suite for assessing the psychological fidelity of generative agents. The evaluation integrates multiple approaches, including longitudinal tracking of psychological responses to life events, psychometric assessments of personality and emotion, and automated ethnographic analysis of agent behavior and social interactions. The authors apply these methods across three agent architectures and three base LLMs, demonstrating their use in assessing psychological coherence and identifying behavioral failure modes in long-running simulations.

Critical strengths
  1. Important research question and comprehensive evaluation. T he paper addresses the important challenge of evaluating psychological coherence in long-running agent simulations. Integrating longitudinal, psychometric, and qualitative assessments provides a valuable evaluation perspective, particularly through the incorporation of ethnographic research traditions.

  2. Informative empirical findings. The comparisons across architectures and models reveal interesting differences in psychological stability and responses to life events. The identified failure modes, including repetitive behavioral loops and collective conspiracy dynamics, offer useful insights into the limitations of long-running agent simulations.

Critical weaknesses
  1. Insufficient validation and analytical contribution of the ethnographic evaluation. The incorporation of ethnographic perspectives is promising, but the reliability and validity of the automated evaluation remain insufficiently established. Furthermore, the analysis primarily reports predefined evaluation scores and illustrative cases, without clearly demonstrating what additional insights the ethnographic approach offers into agents' social behaviors and interactions.

  2. Limited examination of cultural contexts and insufficient support for cultural fidelity claims. Although the paper emphasizes culturally situated agent behavior and introduces simulation environments across multiple regions, the main experiments are conducted in a single setting, with only a separate qualitative analysis of the Kerala simulation. The analysis primarily identifies culturally specific references in generated conversations without independent validation of their cultural authenticity or appropriateness by local participants or cultural experts.

Questions for authors
  • Could the authors provide more details on the experimental and evaluation procedures, including the number of independent simulation runs, sampling strategies, and data aggregation methods across the different evaluations?
  • How was the reliability and validity of the automated ethnographic evaluation assessed?
  • What role do the different cultural contexts play in evaluating psychological fidelity? Were comparable analyses conducted across the different regions?
  • How was the cultural appropriateness of the generated conversations assessed beyond the presence of culturally specific references?
2 comments
AuthorsOpenReview ↗

Response to Reviewer p9ke

We thank the reviewer for their supportive evaluation and for highlighting the importance of evaluating long-horizon psychological coherence, the integration of longitudinal, psychometric, and ethnographic traditions, and the value of the failure modes we surface.

Q1: Experimental, sampling, and aggregation details

We have clarified all experimental and aggregation procedures across Sections 3–4 and the Appendices:

  • Simulation runs & independent shards: Our main factorial design crosses 5 base models × 3 decision logics = 15 conditions. Each condition simulates N = 100 agents split across 5 strictly independent 20-agent shards (75 shards / 1,500 agents total) for 100 two-hour ticks (≈ 12.5 simulated days; 8 ticks/day, 7:00 AM–11:00 PM). The same 100 Sobol-sampled personas (Sec. 3.3) are held fixed across all 15 conditions so model and architecture comparisons are unconfounded by persona resampling variance.
  • Shock assignment: At Tick 40 (Day 6 morning), the Game Master delivers the exogenous shock (AI workplace layoff in the main sweep; phone mugging in the acute-stressor sweep) to exactly 50% of agents (n = 10 shocked, n = 10 retained controls per shard), stratified across workplaces.
  • Psychometric schedule & aggregation: All scales run non-causally (save_to_memory=False). Agents complete the 46-item ESM affect lexicon (containing all 32 DEQ items; App. Table 2) twice daily (every 4 ticks) and BFI-2, SWLS, GHQ-12, and MEMS once daily (9:00 PM). Item responses are aggregated to per-agent daily scale composites, then to condition trajectories and centroids; uncertainty is estimated via shard-level cluster bootstraps across the 5 independent shards per condition (Figs. 5, 8) alongside explicit between- vs. within-agent variance decomposition (Figs. 13–14).
  • Ethnographer sampling: For each of the 15 conditions, we extract complete chronological observation, dialogue, and reflection streams for a fixed stratified subsample of n = 10 primary agents (spanning laid-off and retained roles across shards; 150 full-run agent dossiers total) for cross-family evaluation by Claude Opus 4.6 (Table 1).

W1 & Q2: Reliability, validity, and added analytical insight of the automated ethnographer

  1. Human-ethnographer grounding & ongoing validation: Our rubric was not generated ad hoc by an LLM. We built an interactive interview mode directly into the Concordia Island UI and brought a trained human ethnographer to the lab to interview simulated agents firsthand and examine their action and dialogue logs. Our 12 rubric dimensions and 15 diagnostic probes (e.g., unremarkable routines, register shifts, unspoken emotion) were directly inspired by her observations and domain expertise. Formal benchmark validation comparing the automated ethnographer's ratings against human expert coding is ongoing work as we continue collecting data with the ethnographer.
  2. Automated safeguards for reliability and validity: Rather than holistic LLM-as-a-judge scoring—which suffers from verbosity and self-preference bias—our automated pipeline enforces four structural constraints (Sec. 3.5, App. D): (i) cross-family evaluation (Claude Opus 4.6 never rates Claude agents); (ii) evidentiary primacy, requiring verbatim timestamped log quotes for every claim or marking the dimension INCOMPLETE (dynamically renormalizing weights; Eq. 5); (iii) adversarial counter-evidence, requiring raters to log the "Strongest Evidence Against" each dimension before scoring (App. D.5); and (iv) anti-fluency diagnostic probes (P1–P15), which penalize theatrical LLM prose by requiring ordinary stretches, unstated emotion, and register shifts across social partners—capping hyper-articulate narration in the Legible band (55–74, with 0% of agents reaching Inhabited ≥ 75).
  3. Concrete analytical insight beyond questionnaires: Expanding to 5 models yielded a clear example of what ethnography catches that surveys miss. On psychometric scales alone, Qwen3-30B-A3B looks near-human (BFI ICC = 0.85–0.93, construct validity r = 0.74–0.78, DEQ = 0.76). Yet our ethnographer rated 100% of Qwen3 Rational (29.8) and ESA (22.5) agents as Degenerate (Table 1). Checking the fieldnotes and raw stream telemetry confirmed why: Qwen3 agents lock into self-reinforcing text loops where retained employees repeatedly narrate "preparing" for work all day without ever commuting to their workplaces, driving exact-repeat observation openings from 28% (Minimal) to 40% (Rational) to 56% (ESA) (vs. 15–20% in Gemini 3.8 Flash), cutting dialogue from 31.1 to 4.3 turns/agent, and repeating "brow is slightly furrowed" 80.4 times/agent. Similarly, the Gemma 4 post-layoff micro-task loop, the Rational reflective sink, and the Gemma 3 conspiracy spiral were all surfaced by ethnographic fieldnotes while remaining invisible to Likert trajectories.
AuthorsOpenReview ↗

Response 2

W2, Q3, & Q4: Role of cultural contexts, cross-regional comparison, and calibrating cultural claims

We agree that our original wording ("faithfully mirror real-world cultural geographies") overstated what automated log analysis alone can establish without local human evaluators. We have revised Section 4.5.1, Discussion, and Ethics in three ways:

  1. Expanded cross-regional comparison (Brecksville, Kerala, and new Surulere, Lagos run): Alongside Brecksville, Ohio (Fig. 11) and Alappuzha, Kerala (N = 710 dialogues; Fig. 6), we added a 100-agent Surulere, Lagos simulation (N = 593 dialogues; Fig. 12). Comparing sites shows how regional grounding shapes both everyday social priorities and how agents behaviorally adapt to the Day-6 layoff shock: Brecksville agents confront managers over algorithmic efficiency or retreat into private apartments; Kerala conversations center on aesthetic/philosophical themes (31.8% Arts/Music, 29.9% philosophical) and civic transit (KSRTC); and Surulere conversations center on compound security and marital care (34.2%), exact naira-and-kobo household ledgers (17.5%), NEPA power logs, and informal entrepreneurial adaptation to layoffs (e.g., calculating wholesale flour and shortening margins at Lawanson Market to supply a neighborhood meat-pie kiosk; Fig. 12C).
  2. Assessment beyond surface noun references & calibrated scope: Beyond topical coding, our dialogue-depth classification audits whether exchanges are substantive (55.8% Kerala, 64.8% Lagos) vs. trivial small talk (1.7% and 5.6%), and probes P14/Appropriateness check situational register shifts. Nonetheless, we now explicitly frame these findings as demonstrating contextualized cultural grounding (steering agents away from generic Western defaults into coherent local scripts) rather than certified cultural authenticity, and highlight participatory evaluation with local residents and cultural experts as an essential requirement for validation.
Reviewer_q4LDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Concordia Island, a scalable, open-ended sandbox that simulates over 100 agents' lives and routine activities in a configurable geographical and cultural environment. The authors propose a comprehensive evaluation suite that focus on the synthetic psychology of the simulated agents, covering metrics such as reactions to external shocks, psychometric reliability and validity, and ethnographic analysis on the simulated tracks. The authors reveal that, while LLM agents can reliably simulate emotional responses and individual differences, there exist systematic failure modes across multiple runs and models.

Critical strengths
  1. The authors focus on the psychological and behavioral fidelity of generative agents in social simulation sandbox via synthetic psychology, which is crucial to back the validity of potential insights and conclusions produced from the simulation process. The assessment also covers a broad range of dimensions.
  2. The simulation is not limited to a single environment setting, which allows cross-cultural comparisons of simulation patterns and metrics.
Critical weaknesses
  1. The advantage of Concordia Island compared to other social simulation platforms is not very significant. Although the experiment design has its unique value, and the evaluation suite is genuine novelty, the general design of the sandbox itself seems no different from existing ones. For instance, the authors mentioned AgentSociety, which also simulates on real road networks and covers multiple economic and social classes; they highlight that the difference comes from the problem being studied, however this doesn't limit the simulation itself to Concordia Island.
  2. The evaluated model is limited to the Gemini/Gemma family, which comes from the same company and can potentially lack diversity. For example, the repeated failure modes mentioned in Section 4.5 could be a prevalent issue on this model family rather than a problem across all mainstream LLMs.
Questions for authors
  1. What are the advantages to conduct social simulation experiments on Concordia Island? Can the synthetic psychology studies be conducted on AgentSociety or other sandboxes too, so a cross-platform study is possible?
  2. Have you tried to run Concordia Island with other model families like Qwen or gpt-oss?
  3. One of your preset community settings is Lagos, Nigeria, but there seems no results on this particular setting (compared to the Kerala, India setting in the case study). Does this setting provide any unique findings?
    • Furthermore, is it possible to run multi-cultural communities that allows cross-cultural interactions and clashes?
3 comments
AuthorsOpenReview ↗

Response to Reviewer q4LD

We thank the reviewer for their positive assessment of the paper's clarity, the importance of synthetic psychology for backing simulation validity, and the genuine novelty of our evaluation suite and multi-setting design.

W2 & Q2: Expanding beyond the Gemini/Gemma family (Qwen3-30B-A3B and Gemini 3.8 Flash) Thank you for this suggestion. We expanded our 33 comparison to a 53 factorial study (N=1,500 agents) by adding Qwen3-30B-A3B (Alibaba open-weights MoE, outside the Google ecosystem) and Gemini 3.8 Flash across all experiments (Figs. 3–14, Table 1):

  • Core psychometric & shock patterns generalize: In Qwen3-30B-A3B, Big Five traits are highly stable (ICC=0.85--0.93), trait–well-being cross-correlations match the human meta-analysis (r=0.74--0.78, p<0.01; Fig. 4), discrete emotion congruence reaches =0.76 under ESA (Fig. 5), and job-loss recovery depends on memory architecture (Minimal recovers 67%67\% of its drop vs. 28–40%28\text{--}40\% in Rational/ESA; Fig. 7D1). Meanwhile, Gemini 3.8 Flash ESA achieves our highest overall trait stability (ICC=0.92) alongside strong post-shock scarring (d=3.13; Fig. 3).
  • Do failure modes generalize outside Gemini/Gemma? Yes—and Qwen3 reveals an even sharper one: Rather than Gemma 3's paranoia spiral, Qwen3 exhibits affective inertia on state surveys (lowest within-agent mood fluctuation of all 15 runs, ESMSDwithin=0.17; Fig. 14B) and self-reinforcing template loops in open-ended behavior (Table 1). Even while scoring near ceiling on questionnaires, Qwen3's ethnographic score drops from 34.4 (Minimal) to 29.8 (Rational) to 22.5 (ESA, 100% Degenerate), as added prompt stages recycle stock phrases (56%56\% repeated observation openings; dialogue dropping from 31.1 to 4.3 turns/agent). Thus, long-horizon behavioral attractors are a cross-provider challenge for open-ended LLM agents, though each model family exhibits a distinct failure signature. (Our open-source codebase also supports any OpenAI-compatible endpoint for evaluating gpt-oss and future models.)

W1 & Q1: Positioning relative to AgentSociety and cross-platform applicability We agree that the Synthetic Psychology Evaluation Suite is our primary contribution, and Concordia Island is the controlled testbed built to run it; we have clarified this in the Introduction and Discussion. Two points distinguish our approach from AgentSociety (Piao et al., 2025) and support cross-platform studies:

  1. Depth of individual psychology vs. macro aggregates: AgentSociety validates macro-level urban outcomes (mobility, polarization), while modeling individual minds via coarse categorical tags (Mobility, Economy, Other), single-word labels (Need: Safe or Whatever, Emotion: Resentment), and one-line cognitions ("Sequential task execution ensures workflow integrity", their Table 6). Our work tests the micro-level premise macro-simulations take for granted: whether individual agents sustain reliable personality traits, human-like trait–affect covariance, differentiated adaptation to chronic vs. acute shocks, and multi-turn interpersonal relationships. Furthermore, Concordia Island enforces strict co-location observation filtering via a centralized Game Master (App. A.2), swaps cognitive architectures (Minimal, Rational, ESA) on identical personas, and runs all surveys non-causally (save_to_memory=False).
  2. Cross-platform portability: Yes—our evaluation suite is platform-agnostic. Its three modules require only (i) periodic non-causal questionnaire hooks (without writing to agent memory), (ii) a timestamped exogenous shock hook, and (iii) per-agent action/dialogue/reflection logs in JSON format. It can therefore be attached directly to AgentSociety or other sandboxes to enable standardized cross-platform psychological benchmarking.

Q3: Surulere, Lagos results and multicultural communities

  1. New Surulere, Lagos analysis (Sec. 4.5.1, Fig. 12): We ran and analyzed a 100-agent Surulere, Lagos simulation (N=593 multi-turn dialogues, 4,003 turns; Gemini 3.8 Flash ESA) alongside Alappuzha, Kerala (N=710 dialogues; Fig. 6). Both sustain predominantly substantive conversations (64.8%64.8\% in Lagos, 55.8%55.8\% in Kerala; trivial small talk <6%< 6\%), but surface distinct local adaptations: whereas Kerala centers on classical raag aesthetics, coir weaving, and KSRTC transit, Surulere centers on Compound Sanctuary & Marital Care (34.2%34.2\%), Household Budgets & Kobo Ledgers (17.5%17.5\%), and Market Trade (10.8%10.8\%)—including naira-and-kobo accounting for Agege bread and kerosene, bargaining over Middle Belt egusi haulage at Lawanson Market, tracking NEPA power restoration, and laid-off workers pricing wholesale flour at Lawanson to launch informal meat-pie stalls after Day-6 layoffs.
AuthorsOpenReview ↗

Response 2

  1. Multicultural communities: Yes, Concordia Island supports mixed-culture towns out of the box because cultural background, formative memories, and worldview are instantiated per agent in the persona generator (Sec. 3.3) rather than hardcoded globally into the engine. Populating a shared neighborhood map with personas sampled from multiple regional distributions enables studying cross-cultural interaction, diaspora networks, and norm friction—an exciting direction we highlight for future work.

Thank you for the response and revised manuscript. The Qwen3 experiments address my concern about model diversity. The contrast between strong questionnaire performance and repetitive behavior strengthens the empirical case for the combined evaluation suite.

My reservation about the sandbox’s limited novelty remains. The Lagos analysis also answers my question about that setting, although the different base models used for Kerala and Lagos limit the interpretation of comparisons between them.

I've adjusted my ratings accordingly.

Reviewer_J8V1OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Concordia Island, a simulation environment for studying the long-term psychological behavior of generative agents. The authors simulate populations of 100 agents and compare three agent architectures across three language models. They evaluate the agents using psychological questionnaires, controlled events such as job loss and mugging, and an automated LLM-based ethnographer. The paper studies whether agents show stable personalities, realistic emotional changes, and human-like psychological structure over time.

Critical strengths
  1. The paper studies agents over multiple simulated days. This is more interesting than only asking agents to complete questionnaires once.

  2. The evaluation covers personality, emotion, life satisfaction, external shocks, and qualitative behavior. The paper also compares three agent architectures.

  3. The paper reports several interesting failure cases. For example, some agents develop persistent conspiracy beliefs after ordinary events.

Critical weaknesses
  1. The evidence does not support the claimed recovery of human emotional structure. The Tucker congruence scores are around 0.60 to 0.78, while the paper itself defines 0.85 as the minimum for a fair match.

  2. The shock experiments may mainly measure prompt compliance. An agent reporting lower life satisfaction after being told that it lost its job does not necessarily show a persistent psychological process.

  3. There is no matched human baseline. The paper compares agents with previously reported human statistics, but humans do not participate in the same environment or experience the same events.

  4. The automated ethnographer is not validated by human annotators. It is unclear whether its scores measure realistic behavior or simply reward coherent and detailed writing.

  5. The model comparison is narrow. Gemma 3, Gemma 4, and Gemini 2.5 Flash all come from the same provider ecosystem.

  6. The main idea is not very novel. Long-running agent societies already appear in Generative Agents[1]. Psychometric evaluation of simulated agent populations has also been studied before.

  7. The effective number of independent samples is unclear. Agents interact inside shared groups, so treating individual agents as independent may underestimate uncertainty. [1] Generative Agents: Interactive Simulacra of Human Behavior

Questions for authors
  1. How is this work different from prior works?

  2. How do the authors justify the phrase “recovery of human emotional structure” when every Tucker congruence score is below the paper’s own 0.85 threshold?

  3. How many independent simulation seeds are used? How is dependence between agents in the same group handled statistically?

  4. How can the shock experiments distinguish genuine longitudinal change from an immediate response to being explicitly told about a negative event?

2 comments
AuthorsOpenReview ↗

Response to Reviewer J8V1

We thank the reviewer for their detailed critique and for noting the value of studying agents across multiple simulated days, evaluating multiple psychological dimensions across architectures, and reporting concrete behavioral failure modes.

W1 & Q2: "Recovery of human emotional structure" vs. Tucker congruence below 0.85 The reviewer is completely right, and we thank them for catching this overstatement. Congruence values of ϕ=0.60–0.78\phi = 0.60\text{--}0.78 fall below the conventional 0.850.85 threshold for a fair factor match, so "recovery" was not justified. We have renamed Section 4.4 to "Similarity to Human Emotional Structure" and revised the abstract, contributions list, results, and discussion to report partial similarity. Extending this analysis to all 15 conditions (5 models×3 logics5\text{ models} \times 3\text{ logics}, with 95% shard-cluster bootstrap CIs; Fig. 5) yields ϕ=0.60–0.78\phi = 0.60\text{--}0.78, with ESA highest or tied in 4 of 5 models (Gemma 3: 0.780.78, Qwen3: 0.760.76, Gemma 4: 0.750.75, Gemini 3.8 Flash: 0.720.72 vs. 0.620.62 Minimal). Agents avoid collapsing all negative states into a single distress factor (Fig. 9), yet still blur discrete negative emotions (anger, anxiety, sadness) more than human respondents—establishing clear diagnostic headroom for our benchmark.

W2 & Q4: Distinguishing longitudinal adaptation from immediate prompt compliance We agree that an immediate drop in self-report on its own cannot separate psychological state tracking from prompt compliance. Four elements of our experimental design and new 5-model results test this directly:

  1. In-world events and non-causal questionnaires: The layoff is delivered by the Game Master on Day 6 as a workplace event (termination call/notice and revoked workplace access; App. A.4). Questionnaires never mention the shock, and every instrument executes with save_to_memory=False (Sec. 3.4), so prior survey answers never enter memory to anchor later answers.
  2. Matched within-town controls: Every 100-agent run pairs 50 shocked agents with 50 unshocked control agents living and interacting in the same town (N=750N = 750 per group across 15 runs; Fig. 2, Fig. 7). Control trajectories stay flat across all 12 days without drift.
  3. Recovery depends on memory architecture, not the event text: Across all five models, Minimal agents drop sharply on Day 6 after job loss—proving they register the event—but rebound 56–79%56\text{--}79\% of the way back to baseline once the termination notice scrolls out of their 3-observation context window ("experiential amnesia"). By contrast, Rational and ESA agents retrieve the event from associative memory and remain scarred through Day 12 (recovering only 13–40%13\text{--}40\%, with persistent gaps up to +2.26+2.26 SWLS below controls). Pure prompt compliance cannot explain divergent multi-day trajectories for the exact same event.
  4. Stressor specificity (Job Loss vs. Mugging; Fig. 2B, Fig. 7 bottom row): When agents instead experience an acute discrete shock on Day 6 (Mugging—their phone is stolen, while their job and social routines continue), the exact same Rational and ESA architectures that stayed chronically scarred after job loss recover steadily toward baseline (63–76%63\text{--}76\% recovery in Gemma 4 and Gemini 3.8 Flash). Thus, memory-scaffolded agents do not mechanically output low scores after any negative event; persistence depends on whether the shock disrupts ongoing daily roles and routines.

W3: No matched human baseline Running 100 human participants in a closed multi-day town while subjecting half to controlled job loss or mugging is ethically and practically impossible—precisely the gap social simulation aims to address. Following standard validation practice in social science modeling, we benchmark agents against two forms of human ground truth: (i) comparative stylized facts from large longitudinal human panels where exact timescale matching does not apply (flat unshocked baselines, persistent scarring after unemployment vs. rapid hedonic adaptation after discrete acute events; Lucas et al., 2004; Luhmann et al., 2012), paired with matched within-simulation controls; and (ii) direct quantitative human benchmarks where normative empirical matrices exist—specifically the 15-cell Big Five ×\times Subjective Well-Being meta-analytic correlation matrix (r=0.59–0.88r = 0.59\text{--}0.88, all p<0.05p < 0.05; Anglim et al., 2020; Fig. 4) and human BFI-2 between-person trait variance norms (SDbetween=0.61–1.03\text{SD}_{\text{between}} = 0.61\text{--}1.03 vs. human 0.65–0.850.65\text{--}0.85; Fig. 13). We state this scope explicitly in the Discussion and Ethics sections.

AuthorsOpenReview ↗

Response 2

W4: Validation of the automated ethnographer (and whether it just rewards fluent writing)

Human-ethnographer rubric design, three automated controls, and our new Qwen3 experiment address validation and fluency bias:

  1. Human-ethnographer design & ongoing validation: Rather than inventing criteria with an LLM, we built an interactive interview mode into the Concordia Island UI and brought a trained human ethnographer into the lab to interview simulated agents firsthand and inspect their logs. Our 12 rubric dimensions and 15 diagnostic probes were directly inspired by her observations and domain expertise; formal validation comparing automated scores against human expert coding is ongoing work as we continue collecting data with the ethnographer.
  2. Cross-family rating & verbatim-evidence rule: Rating is performed cross-family by Claude Opus 4.6 (evaluating Gemini, Gemma, and Qwen). Every dimension score must cite verbatim log quotes or be marked INCOMPLETE and excluded (App. D.2), and raters must log the "Strongest Evidence Against" each score (App. D.5).
  3. Rubric explicitly penalizes LLM over-fluency: Probes P1–P3 and P13 (Ordinariness, Thickness) penalize dramatic, hyper-articulate narration by requiring unremarkable routines and unstated emotion. Coherent, polished writing with overt meaning on every turn is capped in the Legible band (55–74), and 0 of 150 rated agents across 15 runs reached Inhabited (≥ 75).
  4. Catching failures that surveys miss: Qwen3-30B-A3B writes fluent reflections and scores near ceiling on BFI reliability (ICC = 0.93) and construct validity (r = 0.74–0.78), yet our ethnographer scored 100% of Qwen3 Rational/ESA agents as Degenerate (22.5/100). Direct log counts confirm why: employed agents loop on "preparing" for work without leaving home (56% exact-repeat observation openings; dialogue cut from 31.1 to 4.3 turns; "brow is slightly furrowed" repeated 80.4 times/agent).

W5: Narrow model comparison

We added Qwen3-30B-A3B (Alibaba open-weights MoE, outside the Google ecosystem) and Gemini 3.8 Flash across all 3 decision logics (N = 1,500 agents total; Figs. 3–14, Table 1).

W6 & Q1: Novelty relative to Generative Agents and prior psychometric LLM work

Generative Agents (Park et al., 2023) ran 25 agents for 2 simulated days and evaluated them on a single crowdworker-ranked subjective question ("believability" across 5 interview prompts, with ablations reusing the main run's frozen memory stream), with zero comparison to empirical psychology studies and no psychometric scales or shock trajectories. Conversely, prior LLM psychometric studies administer standalone questionnaires once to isolated models with no ongoing social world. To our knowledge, our work is the first to jointly evaluate multi-day longitudinal shock adaptation, psychometric reliability/construct validity, and quote-grounded ethnography in interacting agent populations—demonstrating that models can pass standalone survey checks while failing in open-ended life (Table in General Response).

W7 & Q3: Simulation seeds, group dependence, and statistical uncertainty

Each of the 15 conditions simulates N = 100 agents split across 5 independent 20-agent shards (75 independent social worlds total; Sec. 3.1). Agents only observe and converse within their own shard, so shards are strictly independent replicates, while holding the 100 personas fixed across conditions provides a paired counterfactual design across models and architectures. To account for within-shard dependence, we compute uncertainty via shard-level cluster bootstraps (resampling whole shards; e.g., 95% cluster-bootstrap CIs in Fig. 5 and bootstrap SEMs in Fig. 8) and decompose between- vs. within-agent variance directly (Figs. 13–14).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.