acceptodds
Withdrawn from ICLR 2027

The White Elephant in the Room – Historical Analogies for AI Discussions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Research in the fields of Machine Learning (ML) and Artificial Intelligence (AI) has changed in the past years. Financial support, hardware, software, data, and scientific processes have improved substantially. These developments contributed to scientific advances across many disciplines and to broader societal effects. How these events will unfold is still unclear. But gaining a better understanding of this situation and responsible usage of AI technology likely requires an interdisciplinary discourse of informed citizens, scientists, and policy makers. We argue that one obstacle to this discourse is structural: many questions raised by AI are not well captured by the field's conventional evaluation apparatus. Benchmarks report performance on specified tasks and metrics, but broader questions about understanding, downstream effects, measurement validity, and investment require additional ways of reasoning. Since a comprehensive theory of how AI will impact our lives remains the subject of an often polarized debate, we here propose a complementary approach familiar to ML researchers: learning from examples. We explore four historical analogies: Alchemy, Plastic Industry, Free Will, and White Elephants. Each analogy foregrounds a different question that benchmarks cannot answer on their own: The Alchemy analogy asks whether we understand what we have built, the analogy on Plastic asks what it will cost us later, the Free Will example asks whether we measure what we think we measure, and the White Elephants analogy asks whether the investment is worth it. Because these narratives are comparatively accessible, they may help structure discussions across disciplinary and professional boundaries, including discussion of where each analogy holds and where it breaks. Ultimately, this may support a more differentiated discussion of possible trajectories for AI.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
History of Science, society, responsible AI, alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_ZJjROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a complementary approach to investigating how AI impact our lives using four historical analogies such as Alchemy, Plastic Industry, Free Will, and White Elephants and learning from examples. Using these narratives, they help structure the discussions across disciplinary and professional boundaries, including discussions of where each analogy holds and where it breaks.

Critical strengths
  1. The paper is relevant and attempts a complementary approach to investigate the role of AI in society using analogies from the past.

  2. I like the counterarguments in the alchemy section, where the authors talk about competing arguments of Rahimi and LeCun, and the white elephant section uses railways and broadbands as counterexamples where they differentiate between poor financial returns and lasting social value.

Critical weaknesses
  1. Novelty and contribution: The contribution of the paper remains unclear beyond the four already existing theories. Furthermore, several analogies already appear in existing literature. The paper explains each analogy and the history behind it well, but the novelty beyond that seems lacking.

  2. The authors draw on how each analogy could be used to better understand AI's role; however, there is no worked example on how these could be used in real-life examples of AI usage.

  3. The motivation behind choosing these four historical analogies is not systematic. The authors do mention that these are not helpful in all aspects and can be used informally, but I'd request an explanation as to why the authors chose these specific analogies.

  4. Regarding emergent abilities, the authors cite works that show they are an artifact of evaluation metrics. These are useful, but at the same time, it is important to also consider that it is important to consider possible collective abilities (see OpenAI-Hugging Face incident).

Questions for authors

See above.

Reviewer_q8ofOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines how four historical analogies can support discussions of AI beyond benchmark performance. Alchemy highlights the relationship between empirical success and scientific understanding; plastics motivate attention to downstream costs and lifecycle responsibilities; free-will research illustrates the distinction between measurable proxies and broader concepts; and white elephants raise questions about investment and social value. Drawing on historical accounts and existing literature, the authors connect these analogies to contemporary AI debates and discuss several limitations and counterexamples. The contribution is a conceptual synthesis intended to facilitate interdisciplinary discussion. Its support consists of literature-based argumentation and illustrative examples, rather than an original empirical evaluation of its communicative benefits.

Critical strengths
  • The paper organizes four issues that benchmarks alone cannot resolve—understanding, downstream consequences, measurement validity, and investment value—and uses historical narratives to make them relevant and accessible across disciplines.
  • It carefully limits its claims by distinguishing digital content from physical waste, acknowledging measurement traditions in intelligence research, and separating financial returns from social benefits, which makes the overall case more credible.
Critical weaknesses
  • The four analogies largely restate familiar concerns about understanding, lifecycle costs, measurement validity, and investment value, without clearly showing what new insight emerges from combining them.
  • The paper proposes analogies as tools for AI discussion but lacks a concrete application showing how they improve assessment, and it remains unclear how the acknowledged limitations should guide the use or rejection of specific comparisons.
Questions for authors
  • What specific new insight does this paper establish beyond the cited literature?
  • In one concrete AI case, what does the analogy reveal that directly stating the underlying concern does not?
  • What criteria determine whether an analogy is informative or misleading, particularly when alternative historical examples suggest different interpretations?
Reviewer_a4gJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes four analogies for AI that they claim will elucidate our understanding of AI: Alchemy Plastics Free Will White Elephants

In each case, they provide a mixture of questions for the field or assorted associations for historical analogies / AI critiques

Critical strengths

The exercise this paper is attempting could be a valuable one, if done right, by highlighting analogies that are a close fit to AI and then showing that they imply unanticipated or addressed issues that the field should address.

Critical weaknesses

This paper fails entirely to meet its stated goals of promoting "better understanding" and "responsible usage" of AI.

The analogies used to understand AI are either already known and better discussed elsewhere (e.g. alchemy) or only of glancing proximity, as the authors themselves note "of course the plastic waste analogy is misleading". Instead, these are vehicles to restate well know challenges in the field and do not add any understanding. Worse, the paper's argumentation for each reads like a high school free-association essay of "things I've read about AI" and doesn't address any of them seriously. For example, the discussion about White Elephants clearly begs for some quantitative analysis: are the costs justified? This is an addressable questions, but here they are discussed in only the most superficial way.

Questions for authors

I have no questions for the authors.

Reviewer_7isuOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors develop an interpretive framework to surface historical parallels to many of the issues today related to the project of artificial intelligence. They draw lessons learnt from failures in alchemy, plastics, free will research and technological gigantism, eliciting reflection on a better way forward for the field of machine learning.

Critical strengths

In a field overrun by publish or perish incentives and other threats to research integrity, the paper elicits critical reflection in applicative researchers that would otherwise hardly get the chance in academic setting to confront issues deeper than micro-optimisation of benchmarking metrics. By drawing an ad-hoc interpretive framework built on easily understandable empirical findings, the authors intelligently lower the high bar of attention and propaedeutic knowledge that would otherwise be necessary to engage with the same topics when adopting more established frameworks in the philosophy of science, epistemology and political economy.

Critical weaknesses

The paper could be improved by reducing the analogies to three, i.e. getting rid of the Free Will analogy and moving the relevant remarks there to the relevant and underdeveloped paragraphs for other analogies (e.g. moving the discussion on evaluation to the "how much understanding is enough? paragraph").

Also, the paper could be improved through a better discussion of power relations/structures, and the responsibility of researchers to rigour, as part of a privileged class from the richest part of the world, as well as of engagement with the downstream impacts of their research (I'm thinking here e.g. of "Data Science as Political Action" by Ben Green).

Questions for authors
  • What is the added value of the table of the historical periods for plastics development?
  • Why not replace it with a table summarising the key points of the analogies and how they relate to open issues in machine learning/the project of ai?
Reviewer_GLpwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper offers four historical examples to support transdisciplinary discussions on trajectories of AI. The use of examples is inspired by the case-based reasoning paradigm, well-established in the ML community, and connects it to the use of narratives as a form of knowledge, well-established in the social sciences and humanities. The chosen analogies are: Alchemy, as an alternative form of knowledge to modern science; the Plastic Industry, as a real-world, tangible technological innovation which allowed production on an unprecedented scale; Free Will, as a subject of scientific inquiry similarly difficult to define as intelligence; and White Elephants, considered a legendary gift in Southeast Asia which embody both foreseeable catastrophic outcomes and an ideology of technological gigantism.

Critical strengths

Overall, I believe tying case-based reasoning to narratives is a promising way to facilitate transdisciplinary discourse, which is crucial in the fast-paced but fragmented AI research landscape. The four examples highlight important aspects and connect the discussion with relevant historical analogies. By not assuming any background knowledge, it invites readers from all sorts of backgrounds to meet on common ground. Each example is presented clearly, leaving room to focus on different aspects: The relationship between theory and experiment, weighing harmful long-term effects against short-term efficiency gains, the limits of benchmarks for measuring complex human concepts, and the difficulty of appropriately justifying the benefits of investments when their dynamics are governed by self-justifying gigantism.

Critical weaknesses

However, the paper also falls short in addressing some issues for both the technical and non-technical communities.

Technical: While using different examples invites the reader to think critically, treating each example in isolation fails to convey a novel overall argument and offers only established ones, such as the need for better evaluation practices in the ML community. To better address the ML community, the authors should be more precise about how the points relate and how these examples connect to concrete steps the community could take to move towards novel quantitative measures of success or failure of ML systems.

Non-technical: As the authors mention, these four analogies are not the only suitable ones. In particular, I do not think the ones presented take the current political environment sufficiently into account, especially its consequences for the most marginalised in society. The Free Will example also falls short of addressing whether intelligence is a useful concept for framing AI in the first place.

Questions for authors
  1. Could you be more specific about how these examples can lead to concrete steps for evaluating ML systems?

  2. How about removing the Free Will example and strengthening the argument by focusing more on power relations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.