- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Concordia Island, a multi-agent simulation environment built on the Concordia framework, and proposes a comprehensive evaluation suite for assessing the psychological fidelity of generative agents. The evaluation integrates multiple approaches, including longitudinal tracking of psychological responses to life events, psychometric assessments of personality and emotion, and automated ethnographic analysis of agent behavior and social interactions. The authors apply these methods across three agent architectures and three base LLMs, demonstrating their use in assessing psychological coherence and identifying behavioral failure modes in long-running simulations.
Critical strengths
-
Important research question and comprehensive evaluation. T he paper addresses the important challenge of evaluating psychological coherence in long-running agent simulations. Integrating longitudinal, psychometric, and qualitative assessments provides a valuable evaluation perspective, particularly through the incorporation of ethnographic research traditions.
-
Informative empirical findings. The comparisons across architectures and models reveal interesting differences in psychological stability and responses to life events. The identified failure modes, including repetitive behavioral loops and collective conspiracy dynamics, offer useful insights into the limitations of long-running agent simulations.
Critical weaknesses
-
Insufficient validation and analytical contribution of the ethnographic evaluation. The incorporation of ethnographic perspectives is promising, but the reliability and validity of the automated evaluation remain insufficiently established. Furthermore, the analysis primarily reports predefined evaluation scores and illustrative cases, without clearly demonstrating what additional insights the ethnographic approach offers into agents' social behaviors and interactions.
-
Limited examination of cultural contexts and insufficient support for cultural fidelity claims. Although the paper emphasizes culturally situated agent behavior and introduces simulation environments across multiple regions, the main experiments are conducted in a single setting, with only a separate qualitative analysis of the Kerala simulation. The analysis primarily identifies culturally specific references in generated conversations without independent validation of their cultural authenticity or appropriateness by local participants or cultural experts.
Questions for authors
- Could the authors provide more details on the experimental and evaluation procedures, including the number of independent simulation runs, sampling strategies, and data aggregation methods across the different evaluations?
- How was the reliability and validity of the automated ethnographic evaluation assessed?
- What role do the different cultural contexts play in evaluating psychological fidelity? Were comparable analyses conducted across the different regions?
- How was the cultural appropriateness of the generated conversations assessed beyond the presence of culturally specific references?
2 comments
Response 2
W2, Q3, & Q4: Role of cultural contexts, cross-regional comparison, and calibrating cultural claims
We agree that our original wording ("faithfully mirror real-world cultural geographies") overstated what automated log analysis alone can establish without local human evaluators. We have revised Section 4.5.1, Discussion, and Ethics in three ways:
- Expanded cross-regional comparison (Brecksville, Kerala, and new Surulere, Lagos run): Alongside Brecksville, Ohio (Fig. 11) and Alappuzha, Kerala (N = 710 dialogues; Fig. 6), we added a 100-agent Surulere, Lagos simulation (N = 593 dialogues; Fig. 12). Comparing sites shows how regional grounding shapes both everyday social priorities and how agents behaviorally adapt to the Day-6 layoff shock: Brecksville agents confront managers over algorithmic efficiency or retreat into private apartments; Kerala conversations center on aesthetic/philosophical themes (31.8% Arts/Music, 29.9% philosophical) and civic transit (KSRTC); and Surulere conversations center on compound security and marital care (34.2%), exact naira-and-kobo household ledgers (17.5%), NEPA power logs, and informal entrepreneurial adaptation to layoffs (e.g., calculating wholesale flour and shortening margins at Lawanson Market to supply a neighborhood meat-pie kiosk; Fig. 12C).
- Assessment beyond surface noun references & calibrated scope: Beyond topical coding, our dialogue-depth classification audits whether exchanges are substantive (55.8% Kerala, 64.8% Lagos) vs. trivial small talk (1.7% and 5.6%), and probes P14/Appropriateness check situational register shifts. Nonetheless, we now explicitly frame these findings as demonstrating contextualized cultural grounding (steering agents away from generic Western defaults into coherent local scripts) rather than certified cultural authenticity, and highlight participatory evaluation with local residents and cultural experts as an essential requirement for validation.
Response to Reviewer p9ke
We thank the reviewer for their supportive evaluation and for highlighting the importance of evaluating long-horizon psychological coherence, the integration of longitudinal, psychometric, and ethnographic traditions, and the value of the failure modes we surface.
Q1: Experimental, sampling, and aggregation details
We have clarified all experimental and aggregation procedures across Sections 3–4 and the Appendices:
save_to_memory=False). Agents complete the 46-item ESM affect lexicon (containing all 32 DEQ items; App. Table 2) twice daily (every 4 ticks) and BFI-2, SWLS, GHQ-12, and MEMS once daily (9:00 PM). Item responses are aggregated to per-agent daily scale composites, then to condition trajectories and centroids; uncertainty is estimated via shard-level cluster bootstraps across the 5 independent shards per condition (Figs. 5, 8) alongside explicit between- vs. within-agent variance decomposition (Figs. 13–14).W1 & Q2: Reliability, validity, and added analytical insight of the automated ethnographer
INCOMPLETE(dynamically renormalizing weights; Eq. 5); (iii) adversarial counter-evidence, requiring raters to log the "Strongest Evidence Against" each dimension before scoring (App. D.5); and (iv) anti-fluency diagnostic probes (P1–P15), which penalize theatrical LLM prose by requiring ordinary stretches, unstated emotion, and register shifts across social partners—capping hyper-articulate narration in the Legible band (55–74, with 0% of agents reaching Inhabited ≥ 75).