- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates whether the factual reliability of LLMs is primarily a property of an entity's overall popularity or of the specific fact being queried. By holding entities fixed and probing 30 numeric statistics across four domains using seven models, the authors demonstrate that the fact statistic explains roughly half of the variance in answer error, whereas entity popularity explains only about 5%.
Critical strengths
None
Critical weaknesses
The writing is not good enough and requires additional iterations to make the paper clear and human-readable.
The quantification of the fact mentions is not good and does not cover the real mentioning of the facts, which can occur in many different phrasings (unlike entities, which are an acceptable proxy). They can also occur in many documents which are not the main document of the entity (only the head/subject entity?). For example, with "X wrote Y", a mention of this fact can also be in the document about entity Z (e.g., Z was inspired by Y written by X). Also, I might not understand exactly how this statistic is computed because the writing is bad and Appendix C is not human-readable.
Where are R1, R2, and R3 defined?
The coverage of the facts is too limited to draw general conclusions.
The results section reads like a technical report of numbers, without trying to create a narrative, findings, and conclusions, which are the bread and butter of a scientific paper.
The findings are not novel and are pretty trivial, and were shown many times before.
Questions for authors
I recommend rewriting the paper, extending the experiments to cover more facts, and trying to come up with a scientific narrative. You may try to elaborate more on the questions raised in the weaknesses section to increase my understanding of the paper (because I might have missed things because the paper is written poorly), which may result in an increased score.
The novelty is very limited. The paper replaces entity-level frequency with the frequency of an entity’s relations or attributes, yet reaches the same conclusion: more frequent information produces fewer hallucinations. This conclusion is trivial and entirely predictable. Changing the granularity of the frequency statistics adds very little to what was already known. Not to say this comparison is with a paper published in 2023.
I also see little practical value in this approach. Consider a modern agent workflow: this method would itself be another external tool. Before searching for an answer, the system would first need to cache Wikipedia, match the question to the corresponding entity and attribute, and check their frequency to decide whether to search. The answer is already readily accessible through external sources, yet the system uses those sources to decide whether to let the agent answer from memory. When the frequency is high, the agent is still expected to recall the answer and can still hallucinate. This adds machinery around an answer that the agent could simply retrieve