- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the information-theoretic trade-offs among memorizing facts, learning reusable rules, and hallucination under limited model capacity. It begins with the random-fact model of Guo and Li (2026), deriving a lower bound on the false-positive hallucination rate in terms of the factual information retained by the model. The paper then extends this setting to incorporate composite facts generated by reusable rules. By decomposing the information retained by the model, the authors show that rule learning and atomic-fact memorization compete for a shared information budget, leading to a further lower bound on the hallucination rate under perfect atomic recall. Finally, the paper analyzes an additive Gaussian model to illustrate how training-data mixtures affect factual recall and hallucination, and presents synthetic relational experiments to investigate these trade-offs.
Critical strengths
-
Understanding the relationship between memorization, hallucination, and model capacity is an interesting and important research problem.
-
The paper is generally well presented, and the mathematical derivations appear largely sound.
Critical weaknesses
-
My main concern is the limited technical novelty. The random-fact setting and its results, particularly Theorem 3.3, appear to follow directly from Guo and Li (2026). The main conceptual novelty seems to lie in extending the framework to incorporate reusable rules. This could be an interesting contribution in its own right. However, the current technical development relies primarily on a mutual-information chain-rule decomposition and standard information-theoretic arguments, including techniques already used by Guo and Li (2026). It does not appear to provide substantial additional insights that exploit the specific structure of the rule sets or characterize how that structure affects the trade-off between rule learning and factual memorization.
-
The tightness of the main information-theoretic bounds remains unclear. The paper establishes necessary information-budget constraints but does not provide matching achievability results for the rules-and-facts setting, even for specially constructed rule families, let alone realistic learning models. Consequently, it is unclear whether these lower bounds are tight or how accurately they characterize the fundamental trade-offs. In comparison, Guo and Li (2026) provide matching achievability constructions for their random-fact setting. Establishing a similarly tight characterization for the rules-and-facts extension would considerably strengthen the theoretical contribution.
-
The experiments are more illustrative than a direct validation of the theoretical results. In particular, the experiments do not measure or estimate the information quantities introduced in the theoretical framework, nor do they quantitatively test the predicted information-budget constraints. As a result, the observed trends provide only qualitative support for the proposed interpretation, rather than convincing evidence that the theory explains the behavior of the trained models.
Questions for authors
See above
Thank you for your response. I maintain my score.