- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates LLM errors (we could call them hallucinations) in the context of negation. The main takeaway is that by default their outputs change depending on whether a property is mentioned, and whether the mention is negated. This change in the prompt / input leads to different LLM outputs: unknown or not (when answering a question). In other words, LLMs sometime (wrongly “assume” that a property is not true when nothing is stated about that property for a specific entity. The authors work with 14 LLMs
Critical strengths
-
LLMs generally underperform when the input has a negation, and understanding why is certainly important. Building on previous work, the motivation to work on negation is solid.
-
The authors present results with 14 LLMs. That is certainly impressive, but I have to wonder why those 14.
Critical weaknesses
-
Unless I am missing something, the main conclusion is that stating a property holds (for some entity) makes LLMs hallucinate that the same property does not hold for another entity. That seems thin to me for a full paper.
-
Following up with the above, would explicitly stating how to treat negation in the system prompt (or input prompt) solve the problem? For example, what happens if you prompt with “Answer the question below based on the facts below. Do not assume anything except what is explicitly stated.”
-
While the quantitative results are extensive (14 LLMs), what I learned is what these LLM output in the very narrow problem the paper presents. There are no actual causes, but rather some extensive LLM testing (observing outputs given inputs).
-
264: “and pretrained checkpoints already show the lookup”. Where is the evidence for this, and what checkpoints are the authors referring to?
-
What are the chat templates in 297?
The paper is very hard to follow, to the point that it is a major weakness and not just a matter of a few quick edits
-
“both questions look P up in the text, and only the negated one reads a failed lookup as an answer” LLMs don’t look for or at anything, they process the input text (always, regardless of what the input is). LLMs don’t read per se, and if we want to say that processing the input is “reading”, they read whatever is given to them in the input. They don’t choose what to read, even if they appear to ignore a chunk of the input based on observations of their outputs.
-
“Editing the text moves the error where this account predicts: removing the property word largely removes it, a rule that merely mentions the word brings it back, and writing “not P ” into that rule spreads it to the plain question.” I think they mean that LLMs output change if the reverse the polarity of a statement (the statement states a property, based on some examples in the paper), but that wording is extremely convoluted.
These two examples are from the abstract alone. The rest of the paper is not much better. Many terms are undefined or explained in any reasonable detail. For example, the title in 5.1 (“The Lookup Is the Property Word’s Read”) is hard to parse.
The last two paragraphs in Section 2 seem to be a draft with no direct connection to the remaining of the section. “Rule texts” have not been defined. I can’t tell what “open” means in 128.
Here is another example of weird word choices: 144 lists “rule texts, everyday records, and human-written paragraphs alike”. Yet in 156, they state “We wrote 240 records […]”. So the records are also human-written (by the authors)? I find the paper very hard to comprehend.
From the conclusions: “When a model does not find X in a text that mentions P, it keeps the plain question open and judges the negated one true, reading what is not established as established not.” I could guess, but what is X? Keeping a question open means to not answer? That would contradict the paper (the LLM always answer?). “Reading” should probably be rephrased as “assuming”, “making up”, or “hallucinating”.
There are too many examples to list. The paper needs to be rewritten.
Questions for authors
-
Can you justify the need for 14 LLMs, and specifically those? I don’t have a problem, but I do wonder why those.
-
Are you missing a reference in 264 (definition of lookup effect)?
-
See weaknesses. If a question raises to the level of a weakness, I wrote it there.