The Accidental AI Cartel: Toward Behavioral Foundations of LLM Collusion

LLMs are increasingly being deployed in enterprise settings, presenting a shift from siloed, single models to interacting systems of models. Our paper studies collusion, a notable concern in multi-LLM systems. Using auctions as a testbed, we perform a thorough investigation into collusion in economic settings. We first affirm the prevalence and robustness of collusion and find, surprisingly, that LLM collusion is often sub-optimal. We then perform a deep behavioral dive into the nature of LLM collusion, examining which factors influence LLM collusion performance and how LLMs coordinate during collusion. Finally, we consider the engineering of multi-LLM systems. We explore what can amplify, mitigate and detect collusion. Altogether, our paper aims to bolster our foundational understanding of collusion in multi-LLM systems, underscoring the need for continuous evaluations of multi-LLM systems as a whole.