acceptodds
Under review as a conference paper at ICLR 2027

Predicting Where Recall Fails: The Fact Matters More Than the Entity in Black-Box Language Models

Abstract

Language models recall some facts reliably and fabricate others, and the standard predictor of where recall fails is entity popularity, which assigns one score to every fact about an entity. We ask whether factual reliability is a property of the entity or of the fact. Holding the entity fixed, we probe multiple numeric facts about the same entity and period in four knowledge domains (Major League Baseball, the National Hockey League, national economic indicators, and corporate finance), covering thirty statistics and seven models from six laboratories. The fact matters more than the entity. In the sports domains, the statistic being asked explains about half of the variance in answer error and the player about 5%. Entity popularity has no held-out predictive value, whereas prose redundancy, the Wikipedia mention rate of a statistic computed before any query, reaches a held-out of 0.41–0.44 and predicts error on statistics withheld from fitting ( of 0.33–0.40), where features of the numbers themselves, such as their dispersion and magnitude, do not. Ordering statistics by prose redundancy places the least mentioned tier lowest in all four domains (fact-level exact permutation test, combined ); deviation correlation falls from +0.894 to +0.163 in baseball and from +0.990 to +0.853 in national statistics. A second pre-query measure, reconstructibility from co-occurring facts, indicates which low-redundancy facts are consistent with inference from headline statistics. For the least mentioned baseball statistics, models fabricate values while refusing fewer than 1% of questions, whereas for a season beyond their training data the non-answer rate rises to 24.8%. For two frontier models, the spread of repeated samples locates this boundary without ground truth, with an AUC of 0.952. The assessment needs only public text and API access to the model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, factual recall, hallucination, entity popularity, pretraining data frequency, knowledge boundaries, black-box evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_panwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether the factual reliability of LLMs is primarily a property of an entity's overall popularity or of the specific fact being queried. By holding entities fixed and probing 30 numeric statistics across four domains using seven models, the authors demonstrate that the fact statistic explains roughly half of the variance in answer error, whereas entity popularity explains only about 5%.

Critical strengths

None

Critical weaknesses

The writing is not good enough and requires additional iterations to make the paper clear and human-readable.

The quantification of the fact mentions is not good and does not cover the real mentioning of the facts, which can occur in many different phrasings (unlike entities, which are an acceptable proxy). They can also occur in many documents which are not the main document of the entity (only the head/subject entity?). For example, with "X wrote Y", a mention of this fact can also be in the document about entity Z (e.g., Z was inspired by Y written by X). Also, I might not understand exactly how this statistic is computed because the writing is bad and Appendix C is not human-readable.

Where are R1, R2, and R3 defined?

The coverage of the facts is too limited to draw general conclusions.

The results section reads like a technical report of numbers, without trying to create a narrative, findings, and conclusions, which are the bread and butter of a scientific paper.

The findings are not novel and are pretty trivial, and were shown many times before.

Questions for authors

I recommend rewriting the paper, extending the experiments to cover more facts, and trying to come up with a scientific narrative. You may try to elaborate more on the questions raised in the weaknesses section to increase my understanding of the paper (because I might have missed things because the paper is written poorly), which may result in an increased score.

Reviewer_1eMeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method for testing the factuality of LLMs. The approach essentially involves selecting a set of athletes, countries, and companies, querying seven different models for various numerical data, and then verifying these figures against official records. Additionally, the study analyzes Wikipedia articles to determine which metrics are frequently mentioned and which appear only rarely.

Nowadays agents use tools and with web search enabled, they can access content from sources like Wikipedia, effectively eliminating hallucinations. I feel this paper has no practical significance as of 2026

Critical strengths

It would be an interesting finding if it was submitted to ICLR 2023

Critical weaknesses

I get the impression that similar work already existed back in 2023. For instance, the authors cite papers like https://arxiv.org/abs/2211.08411, https://arxiv.org/abs/2212.10511, and https://aclanthology.org/2023.emnlp-main.557/, yet it’s entirely unclear what, if anything, is actually new here. While the models used in the experiments are quite recent, the work lacks any real practical significance since they simply test for hallucinations without allowing web search

The writing is poor, with large blank sections. For example, lines 077–082 are entirely empty.

Questions for authors

See above

1 comment

The novelty is very limited. The paper replaces entity-level frequency with the frequency of an entity’s relations or attributes, yet reaches the same conclusion: more frequent information produces fewer hallucinations. This conclusion is trivial and entirely predictable. Changing the granularity of the frequency statistics adds very little to what was already known. Not to say this comparison is with a paper published in 2023.

I also see little practical value in this approach. Consider a modern agent workflow: this method would itself be another external tool. Before searching for an answer, the system would first need to cache Wikipedia, match the question to the corresponding entity and attribute, and check their frequency to decide whether to search. The answer is already readily accessible through external sources, yet the system uses those sources to decide whether to let the agent answer from memory. When the frequency is high, the agent is still expected to recall the answer and can still hallucinate. This adds machinery around an answer that the agent could simply retrieve

Reviewer_hD6WOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether factual recall reliability in black-box LLMs is better explained by entity popularity or fact-level characteristics. The authors evaluate seven LLMs across four domains (MLB, NHL, national economic indicators, and corporate finance), probing multiple numerical facts about the same entities. They introduce prose redundancy, measured through Wikipedia mentions of statistic types, and reconstructibility from related statistics to characterize factual recall. Their experiments show that statistic identity explains substantially more variation in recall errors than entity identity, with lower-redundancy statistics generally exhibiting poorer recall. The paper also examines fabrication behavior and explores repeated-sample dispersion as a signal for identifying knowledge boundaries without ground-truth answers.

Critical strengths
  1. I find the paper's central question interesting and well motivated. Previous studies have largely focused on entity popularity, whereas this work investigates why LLMs can recall some facts about the same entity much more reliably than others.
  2. I appreciate the breadth of the evaluation, which covers seven models and four different knowledge domains. The authors also conduct several analyses to examine alternative explanations.
Critical weaknesses
  1. Mismatch between the claimed fact-level analysis and the actual redundancy measure While the paper provides convincing evidence that recall reliability varies substantially across statistics within the same entity, I am concerned that its proposed fact-level redundancy measure does not fully capture the exposure of individual facts. Specifically, the measure counts mentions of statistic names in Wikipedia articles rather than occurrences of the actual entity–attribute–value relationships being queried. In the main predictive analysis (Table 1), a single redundancy score is assigned to each statistic type across all entities, regardless of whether the specific numerical facts are documented. The results primarily establish a relationship between statistic-level prominence and aggregate recall difficulty, rather than demonstrating that the exposure of individual facts predicts their recall reliability. This distinction matters because the paper motivates its contribution through fact-level knowledge boundaries and pre-query reliability estimation. Without disentangling statistic-name frequency from the actual exposure of individual facts, I find the interpretation of the proposed measure and the scope of the conclusions insufficiently justified.

  2. Insufficient validation of practical predictive utility The paper demonstrates that fact-level redundancy predicts recall errors more effectively than entity popularity. However, I find its practical utility insufficiently established. A central motivation is to estimate factual reliability before querying an LLM, yet the experiments primarily evaluate statistical associations and held-out prediction performance rather than whether these predictions lead to better decisions in downstream question answering. A higher predictive R2R^2 does not necessarily translate into more reliable identification of questions requiring external verification or abstention. Without evaluating how entity-level and fact-level predictors affect such decisions under comparable budgets, it remains unclear whether the proposed approach offers meaningful practical benefits beyond improving the statistical characterization of recall errors. I believe this leaves an important gap between the paper's predictive motivation and its demonstrated utility.

  3. Limited validation of knowledge-boundary detection. The reported AUC of 0.952 is impressive, but I am concerned that it does not directly measure the proposed method's ability to identify unreliable factual answers. Specifically, the evaluation uses high- versus low-redundancy tiers as classification labels, rather than whether the model's answers are actually correct or fabricated. Although these categories exhibit different aggregate recall errors, they are not equivalent to individual answer reliability. A model may consistently produce an incorrect answer, while a low-redundancy fact may still be recalled accurately. As a result, the high AUC demonstrates that response dispersion separates two predefined fact categories, but does not establish that it can reliably detect factual errors. Moreover, the result is obtained from only 159 MLB question cells and two frontier models, with substantially weaker discrimination between the adjacent R3 and R2 tiers (AUC = 0.590). These limitations make the broader claim of knowledge-boundary detection less convincing than the reported AUC initially suggests.

Questions for authors

See weakness above.

Comments on the paper

3 comments
Area_Chair_9R8UOpenReview ↗

Reminder

Dear reviewers,

Please review the authors’ responses carefully and discuss with the authors if you have more concerns or questions. If their responses have changed your assessment, please consider revising your scores accordingly.

Thank you for encouraging the reviewers to engage with the authors. I really appreciate you doing this as an AC, especially since many ACs do not take this step.

I have already responded to the authors. I believe my original assessment was fair, and their response has not changed my assessment.

Fair enough. Thank you so much for the effort! I'll carefully consider your opinions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.