The White Elephant in the Room – Historical Analogies for AI Discussions

Research in the fields of Machine Learning (ML) and Artificial Intelligence (AI) has changed in the past years. Financial support, hardware, software, data, and scientific processes have improved substantially. These developments contributed to scientific advances across many disciplines and to broader societal effects. How these events will unfold is still unclear. But gaining a better understanding of this situation and responsible usage of AI technology likely requires an interdisciplinary discourse of informed citizens, scientists, and policy makers. We argue that one obstacle to this discourse is structural: many questions raised by AI are not well captured by the field's conventional evaluation apparatus. Benchmarks report performance on specified tasks and metrics, but broader questions about understanding, downstream effects, measurement validity, and investment require additional ways of reasoning. Since a comprehensive theory of how AI will impact our lives remains the subject of an often polarized debate, we here propose a complementary approach familiar to ML researchers: learning from examples. We explore four historical analogies: Alchemy, Plastic Industry, Free Will, and White Elephants. Each analogy foregrounds a different question that benchmarks cannot answer on their own: The Alchemy analogy asks whether we understand what we have built, the analogy on Plastic asks what it will cost us later, the Free Will example asks whether we measure what we think we measure, and the White Elephants analogy asks whether the investment is worth it. Because these narratives are comparatively accessible, they may help structure discussions across disciplinary and professional boundaries, including discussion of where each analogy holds and where it breaks. Ultimately, this may support a more differentiated discussion of possible trajectories for AI.