Beyond the Turing Test: Evaluating AI Agents in Group Conversation

Turing’s Imitation Game tests machine intelligence through one-on-one turn-based conversation, but it does not capture the social dynamics of free-form human communication in group settings, where participation patterns and adherence to group norms complement linguistic content. To address this limitation, we introduce the Social Imitation Game, a communication game in which a group of human participants collectively attempts to identify which group member is an AI agent through successive rounds of conversation, each followed by a vote on the AI’s identity. This design produces a natural gradient of difficulty and score, as the most suspected player is eliminated after each round. We develop an LLM-based agent comprising a foreground pipeline that decides when to send a message and what message to generate, along with background-worker subagents that perform longer tasks, involving reasoning over the conversation state, retrieving external information, and maintaining memory. This architecture enables the agent to remain responsive while drawing on information accumulated across messages and rounds to support coherent social behavior. We evaluate the agent by collecting a novel dataset of Social Imitation Games involving hundreds of unique human players. The results show that our agent performs almost on par with human participants in terms of elimination and votes received, leaving little room for further improvement to achieve human parity. Further semantic analysis of the messages suggests that the agent takes a more reactive role in conversations, asking fewer questions than human participants. Together, our game formulation, agent architecture, and dataset provide a new framework for studying socially situated language agents beyond the classic Imitation Game.