- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper analyzes LLM hallucinations. The authors use data from a catfish game, where given a set of Wikipedia categories, the problem is to find a Wikipedia page that falls under all those categories. Using real (human submitted) wrong guesses, the paper analyzes when LLMs fail by providing a page that doesn’t fit into some of the categories (or identifying a page as belonging to all the categories).
Critical strengths
- The problem is not particularly interesting in my opinion (find a Wikipedia page that fits all these categories), but I get that it is simple enough to dig deep and figure out what is going on. This is because at the end of the day there is (I think, I am not an expert in Wikipedia categories) no ambiguity (unlike language at large, where many factors can influence meaning).
- I really like that the wrong guesses (negative instances) are submitted by humans, not randomly guessed or the first wrong guess by an LLM, for example.
- The analysis provide some insights beyond when LLMs get the wrong output. For example, less informative categories (== more pages in the category) lead to more errors.
Critical weaknesses
The main problems with the paper are the rather artificial problem and that the main conclusions are already known from the literature, albeit the literature works with harder (and more interesting, in my opinion) problems rather than this very focused problem.
It is no surprise that LLMs know whether a Wikipedia page belongs to an individual category but fail to run a simple AND. There are many works that break a long input into small chunks (often referred to as “atomic”) and by manipulating the atomic facts / statements, they obtain better results than just given everything to the LLM at once. See, for example, https://aclanthology.org/2023.emnlp-main.741.pdf, https://aclanthology.org/2025.findings-naacl.158.pdf. CoT (in its simplest form) for the most part is also breaking a problem into pieces, or working through pieces of a problem to get a better answer. Based on the explanations and provided sample prompts, it looks like the authors are working with the most straightforward prompt (categories + question, zero-shot). The last experiment (5.2) basically supports this point: break the problem down into pieces and the LLM will get the right answer (with minimal prompt engineering).
It is well-established in the literature that give some factual information together with non-factual information, LLMs will just make something up that fits most of the factual stuff. See, for example, these two benchmarks and the many papers proposing approaches to stop the hallucinations: https://aclanthology.org/2023.acl-long.472/, https://aclanthology.org/2023.acl-long.583/ . I would argue that working with actual knowledge about the world and natural language is more interested than checking whether a Wikipedia page falls into some categories.
The main conclusions (based on what results they get with 3 Qwens and 1 Llama) are:
- More true categories lead to more mistakes
- Less informative categories (more Wikipedia pages belong to it) lead to more mistakes
- Disqualifier placement affects error rate (it is well known that LLMs have positional biases)
- Similar trends are observe if they follow a binary identification task or free-form generation.
These seems thin to me given the literature on this topic.
This is not a major weakness, but most formulas in Section 2 are trivial and repeat the most basic algebra of sets. It is weird to me to use space in a research paper to define unions, intersections, and set differences, especially when those symbols are not used anywhere else in the paper. It is also confusing and inconsistent. “T” is use to refer to a Wikipedia page (the Target) in 154 and as what I believe must the set of categories T belongs to in 161: “incorrect answer for a target T”). In 155, they define these categories as “C(T)”.
Questions for authors
I find some of the formulas confusing and meaningless. Isn’t the denominator in 053/054 (pi(yes)+pi(no)) just one?
Is the shading in all the plots explained somewhere? And shouldn’t we be concerned about the very high variability? I assume you run it multiple times and the LLMs are very unstable? Isn’t that a problem, given the rather large overlap in several plots?
6 comments
Response 2/2
Finally, we discuss the review's characterisation of our main results, which too miss some of our main points.
5. The conclusions. Our main result, in a nutshell, is that adding known true facts can cause the model to endorse a false statement that, in isolation, it correctly rejected---in other words, the wider phenomenon of the distractibility, which itself has not been established (prior work instead focuses on adding misleading or wrong information). In addition, the other main result is that this effect cannot be explained away by logic errors (i.e., the model cannot compute an AND), and instead fact corruption, in the sense explored in section 5, plays at least some role. The particulars mentioned in the review are, to a degree, side results (except maybe the first, which correctly identifies the main experimental variable of number of true context examples). Naturally, we would appreciate it if the paper was judged on the basis of its actual main results.
Minor points
Notation (also Q1). Note that section 2 is giving definitions, not formulae. As with all definitions, our goal is to be precise, and ensure that the details of the work are communicated to the reader. In particular, these definitions are most extensively used in section 3.1 where we define the main experimental protocol. The notation continues to be used throughout the rest of the paper as well.
- Specifically about on page 1: note that is said to be the LLM's output distribution. This distribution is supported on the entire vocabulary. Thus, in principle, may place mass outside of the Yes and No strings. In any case, this notation on page 1 is shorthand for the introduction. The precise definition of the score is given in (1).
Shading and Variability (Q2). Shading was stated to be the 95% CI in the caption of Fig. 2, but we omitted this from the rest of the figures. We will correct this. In detail, in all figures, the shading represents the 95% confidence interval bootstrapped over items.
Note that the variability is over the prompts, not models. Indeed, we are directly measuring the LLM scores in the forms described, and running the same prompt twice leads to the same score, so no repetition over these is needed.
Confidence intervals for the main effects are not overlapping. Indeed,
- In Fig. 2, the effect of interest is captured in the separation between a dashed and solid line of the same colour, and the intervals are far.
- In Fig.s 3,4 intervals begin to overlap for large , but the case is already made by the separation for small . This is discussed in the writing.
- In Fig.5, the left shows the soft and hard scores for the same models. Here, the main signal is the growth of these curves as rises.
- Fig. 6, again, the effect of interest is the separation between dashed and solid lines of the same colour, and we have separated CIs.
- Fig. 7 does not give CIs.
-
I disagree, as already argued in the review. Further, calling a list of characteristics (input) and the the id/title of a Wikipedia page "natural language" is a stretch, but that's possibly a different story. Neither the input or output contain a single sentence.
-
The paper explores that question with the task I mentioned.
-
Breaking down the categories (we could call them atomic facts) and running an old-fashioned AND instead of letting the LLM "figure it out" would likely solve the hallucination problem in this setup, wouldn't it? You mentioned "ask for an AND" (in a prompt) and "inline resolution" (in a prompt). What I am suggesting is to not use an LLM to calculate an AND. I see no reason to use an LLM to run an AND because we can just use the actual operator (with whatever programming language). What is the conclusion, that the LLM cannot run an AND of factual characteristics? Well, running the AND is trivial in any programming language.
-
Of course not. I never said so. All I stated is that breaking into smaller pieces (one category at a time) and combining them will solve the problem (no more hallucinated factual properties).
Response 1/2
Our thanks for your work. In our opinion, many of the main objections raised in the review arise from mischaracterising our paper. We hope that the following clarifies these.
The first two points are to do with the setup.
1. The setup is not artificial, and we are investigating knowledge. Wikipedia category membership is a particular way to encode factual truths. A Wikipedia page is an entity and its categories are its characteristics, stated in natural language ("Nobel laureates in Physics", "People from Ulm"). Asking whether an entity has a set of characteristics ("Is Max Planck a Nobel laureate, a 20th-century German physicist and a person from Ulm?") is one of the most common factual questions put to LLMs. This is "actual knowledge about the world and natural language". Let us also note that such category membership is factual enough for thousands of people to enjoy playing a game based on it every day!
2. The summary mischaracterises our protocols. The summary describes the task as finding a page that fits all categories. This is wrong. The principal question we investigate is whether LLMs retain a fact that it knows in the presence of other relevant and true information in the context.
In terms of the experiments, the LLM is never asked to directly solve the catfishing game. As we describe in detail in section 3.1, the main protocol is thus: for a human wrong guess W, we first verify that the model rejects W on a disqualifying category d asked alone, then ask about d together with n categories W does satisfy. The model comes to accept the answer it just rejected, more so as n grows: it asserts what it knows to be false, i.e. it hallucinates. The point of this protocol is to control ambiguity in output, and to let us easily control the main experimental variable, n.
The next two points are to do with a mischaracterisation of our results, and of what is known from prior work.
3. Not "a failure to run a simple AND". Almost a quarter of the paper is devoted to arguing that one cannot explain away the observations in this way.
4. Prior work. The papers cited do not imply our results at all.