- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces the Social Imitation Game, a multi-party extension of the Turing Test in which an AI agent participates in live group conversations and human participants repeatedly vote to identify and eliminate the AI. It further develops a real-time agent architecture that separates latency-sensitive message scheduling/generation from asynchronous background processes for social-state tracking, memory, retrieval, and behavioral adaptation. The system is evaluated in 50 live games with 234 human participants; the agent's survival and received votes are close to chance-level baselines, while behavioral analysis identifies some remaining differences between the agent and humans.
Critical strengths
A compelling evaluation setting. The Social Imitation Game moves evaluation from static or dyadic judgments to live, asynchronous multi-party interaction with real humans. Repeated voting and elimination give a natural graded outcome rather than relying exclusively on an offline LLM judge. This is a meaningful evaluation contribution for socially situated agents.
Strong ecological validity and unusually good system documentation. The study evaluates an actually deployed agent interacting with hundreds of human participants rather than only simulated interlocutors, while the implementation, prompts, runtime configuration, and behavioral analyses are documented in substantial detail. This makes the work potentially useful as a dataset and experimental framework beyond the specific reported agent.
Critical weaknesses
The architectural contribution is not empirically isolated. The main result demonstrates that one extensively engineered system is difficult for humans to identify, but there is no sufficiently strong baseline showing what is gained by the proposed foreground/background architecture. In particular, comparisons against a simpler prompt-only agent, foreground-only system, or variants without individual background components would be essential for determining whether the architecture contributes meaningfully beyond the strong backbone model and extensive behavioral prompting. The paper itself acknowledges that the effect of the background workers is difficult to quantify and supports their contribution mainly through qualitative examples.
The broader capability claim is stronger than the experimental scope supports. The evaluation uses one fixed final agent configuration, one main backbone, relatively short games, and a protocol that was preceded by substantial prompt engineering and pilot development. Thus, the results convincingly establish that this particular engineered system can be difficult to detect under this particular protocol, but provide substantially weaker evidence that current LLM agents generally exhibit human-level social behavior in group conversation. The authors themselves appropriately acknowledge that the experiments do not establish architectural optimality.
Questions for authors
Assessment-critical: How does the final system compare against minimally simpler baselines using the same backbone—for example, a single-prompt agent, the foreground pipeline without background workers, or an agent without the Reviewer? Even a limited controlled comparison would substantially affect my assessment of the architectural contribution.
Clarification: How extensive was the development process used to tune the anti-detection behavior, including the number of pilot games and iterations informed by human feedback? Relatedly, is there evidence that the main result transfers across different backbone models, personas, or interaction durations rather than depending strongly on the final tuned configuration?
6 comments
2. Development Process and Potential Overfitting
We understand the concern that extensive development and tuning may limit generalization. Importantly, development and final evaluation involved separate participant populations. Development relied on a small, relatively homogeneous group of in-house testers, whereas the final evaluation included 234 previously unseen participants from 37 countries, recruited through Prolific. Participants ranged in age from 18 to 70, and 155 of 215 with available student-status information were non-students. This demonstrates performance beyond the development population, reducing concerns about overfitting to those participants.
The development phase focused mainly on system reliability and improving natural conversational behavior, including timing, repetition, and response selection, through debugging, prompt refinement, and human feedback. Development involved extensive iterative pilot testing with a small group of in-house testers, including 12 documented Generator–Reviewer configurations, separate from the 50 final evaluation games.
The 2.5-minute round duration and main game settings were chosen in advance for participant experience and retention, rather than to maximize agent performance. We will clarify the development process, its separation from the final evaluation, and these design choices in Section 5.1.
3. Generalizability and Scope of Claims
We appreciate the reviewer's question about whether our results generalize beyond the final tuned configuration. Our contribution is to extend Turing's original Imitation Game to live group conversations, where appearing human requires more than generating convincing text. An agent must also decide when to speak, whom to respond to, and how to adapt to evolving group dynamics. Our results demonstrate that a carefully designed real-time agent can blend into such conversations under the tested conditions.
As discussed above, the final evaluation involved a diverse population of previously unseen participants, providing evidence of performance beyond the development population. However, we have not systematically evaluated generalization across backbone models, personas, or interaction durations, and therefore do not claim that these results necessarily transfer to those settings.
Regarding interaction length, 2.5 minutes refers to a single discussion round, not the entire game. To win, the agent must survive until the final round, not just blend in for a single 2.5-minute round. In games lasting 3–4 rounds, this means remaining indistinguishable from humans throughout a continuous conversation of 7.5–10 minutes. As the game progresses, participants retain the full chat history, accumulate evidence, and can reassess their suspicions based on the agent's behavior over time. Median discussion time was 7.5 minutes, with separate medians of 50 total messages and 11 agent messages.
We will revise the abstract and conclusion to better distinguish our demonstrated contributions from broader generalization claims.
1. Requested Baseline Comparison
We thank the reviewer for the detailed feedback. We address the concerns below, including a new live ablation study comparing our full agent against a reduced version of the architecture.
To address the request for a baseline, we conducted an additional live ablation study consisting of 25 games with all six background workers disabled, compared with 50 full-system games. Both retained Gemini 2.5 Flash and the Scheduler–Generator–Reviewer pipeline, allowing us to assess the added value of the background layer.
Turning off the background agents led to a drop of 10% in games won. The full system won 30% (15/50), versus 20% (5/25) without background workers, and normalized survival was similar (0.548 vs. 0.547). A clearer difference emerged in social participation: with workers, AI and humans each averaged 3.84 messages per active player-round; without workers, AI averaged 3.68 versus 4.14 for humans. The gap was +0.02 with workers versus −0.51 without (difference: 0.53 messages; 95% game-clustered bootstrap CI [0.09, 0.94]). Message length also matched humans more closely with workers (with 4.44 vs. 4.59 words; without, 3.80 vs. 4.60). This suggests workers may help the agent calibrate its participation to the room.
The background worker-off games also reveal specific failures that the background workers are designed to address. In one worker-off game, the agent failed to answer a question about the 2026 World Cup, responding "no idea tbh". Two participants cited its responses as suspicious, and the agent was eliminated. The Internet Context worker could potentially have helped by retrieving up-to-date information. In another worker-off game, the agent repeatedly gave generic agreement, prompting a participant to remark "he is validating anything i guess". The agent was also eliminated. The Repetition Monitor and Behavior Adapter are designed to identify such patterns and encourage more varied, context-appropriate responses. Importantly, Table 3 shows these mechanisms operating in the full system.
These results help clarify what the background workers add to an already capable foreground pipeline. We observe better alignment with human participation and a higher, though statistically uncertain, win rate. The examples illustrate both relevant failure modes and how worker outputs inform foreground decisions. We will add the ablation results and their limitations to the revised paper.