- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether a language model that fails to retrieve a fact still holds partial information about the answer, by analogy with tip-of-the-tongue states in people. Items come from eight Wikidata relations (857 items, later enlarged to 2,163). Three 4-bit models (Mistral-7B-Instruct, Qwen2.5-7B/14B-Instruct) first answer each question. A cross-model knowledge index K(x) then splits failed trials into two groups. Candidate partial-retrieval failures are items where at least one model succeeds, and low-knowledge failures are items where none does. After a failure the model answers up to seven multiple-choice attribute probes, and the Partial Retrieval Score (PRS) is the fraction correct.
PRS is 0.539 for candidate failures and 0.507 for low-knowledge failures (p = 1.2e-3, Cohen's d = 0.19). The gap holds on label-stable trials. It shrinks but survives domain fixed effects, with the coefficient going from 0.174 to 0.118. Each attribute-level lead fails later checks. The pre-registered target, geographic region, does not replicate, and length bucket fails a re-draw, a forced-choice probe and a hidden-state probe. Condition labels agree across reruns only 89.5% of the time despite greedy decoding. In the cue intervention a correct cue helps both groups, with no significant group-by-cue interaction (p = 0.998). The authors narrow their claim to the aggregate PRS effect.
Critical strengths
- The paper is unusually honest about its own results. The authors pre-registered a target, powered a follow-up for it and report plainly that it failed. They then checked their second lead three more ways and report three nulls. They also show that condition labels agree on only 864 of 965 item-model pairs across reruns. The paper points to batch composition and 4-bit quantization as the likely cause but does not test this. Even so, it is a useful caution for work that labels whether a model knows a fact. For me this negative-results value is the main reason to read the paper.
Critical weaknesses
-
The one claim left standing is not separated from entity prominence or from guessing based on priors. By construction, candidate failures are items some model can answer, so this group leans toward more prominent entities. Appendix I lists "partial information disappears after controlling for question difficulty and answer frequency" as falsification criterion 2. Yet the aggregate PRS is never tested against a popularity covariate or a prior-matched guessing baseline. In Table 2 the two largest per-attribute gaps are length bucket and geographic region, the two leads the paper itself shows to fail follow-up. The aggregate gap may therefore rest on the same signal, which is not target-specific.
-
The abstract says the effect "replicates robustly", which goes beyond the evidence. The quoted p = 1.2e-3 comes from the 857-item sample itself. On the 1,310 newly drawn items in §10.1, the gap is 0.543 versus 0.530 (p = 0.130), much smaller than in the original sample. With d = 0.19 and no PRS reported for trials where retrieval succeeded, I cannot tell how much of the attainable signal this gap represents.
Questions for authors
- Does the aggregate gap survive a popularity covariate such as Wikidata sitelinks, and a baseline that always picks each relation's most common attribute value?
- What is the gap after dropping length bucket and geographic region? And after also dropping category, domain and occupation, which Table 2 shows are high and nearly identical in both groups?
- What PRS do trials with successful retrieval reach, and what does a question-only baseline give per attribute?