What Hallucinations Have We Measured? An Empirical Study

Hallucination remains one of the most critical challenges in deploying large language models (LLMs) reliably and safely. Yet the term is used inconsistently across the research community. Analyzing hallucination-titled papers from major NLP and ML venues, we find that explicit definitions are increasingly rare and that the term conflates fundamentally different error types while failing to distinguish hallucinations from expressed uncertainty. This ambiguity obscures what is being measured and why models fail, limiting progress in detection and mitigation. In this work, we show that hallucinated responses should be distinguished from expressed uncertainty and that hallucination is not a unitary phenomenon. We identify two types of model incorrect responses: confabulation, where a model consistently produces incorrect responses, and semantic illusion, where the response correctness varies with the prompt. These two types exhibit distinct empirical signatures and are systematically conflated in existing evaluations. We formalize this distinction with operational definitions and diagnostic criteria, and show that separating them improves interpretability and reveals limitations of current detection methods. We advocate for future work to explicitly distinguish and report these error types.