acceptodds
Withdrawn from ICLR 2027

When Retrieval Fails: Structured Partial Information in Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language-model evaluation typically treats factual retrieval as binary: correct or not. Human memory research instead distinguishes successful recall from tip-of-the-tongue (TOT) states, where partial information about an unretrievable target remains accessible. We ask whether language models show a computational analogue, and state our result up front: we do not establish that models retain target-specific information after retrieval failure. We establish a weaker phenomenon – candidate partial-retrieval failures show greater aggregate accessibility to answer attributes than matched low-knowledge failures, via a Partial Retrieval Score (PRS) over independent attribute probes. Across three open-weight 7–14B models and two natural-item samples (up to 857 items), this aggregate effect replicates robustly () and survives restriction to label-stable items. But every attribute-level lead we identified – geographic region, then character length – fails independent replication, a forced-choice specificity probe, and pre-cue hidden-state probing, and run-to-run label agreement is only under quantized greedy decoding. A causal cue-intervention experiment shows a correct-value cue helps retrieval equally in both groups (the -vs- interaction is null) – evidence that cue content is useful once supplied, not that it was already present internally. We treat the aggregate PRS effect as the paper's one claim to survive scrutiny, and attribute-level or internal-state claims – including our own – as requiring more than a single pilot to trust.

Primary area:
interpretability and explainable AI
Keywords:
tip-of-the-tongue, language models, partial retrieval, factual recall, LLM evaluation, hidden states, replication, cue intervention

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ETvAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a language model that fails to retrieve a fact still holds partial information about the answer, by analogy with tip-of-the-tongue states in people. Items come from eight Wikidata relations (857 items, later enlarged to 2,163). Three 4-bit models (Mistral-7B-Instruct, Qwen2.5-7B/14B-Instruct) first answer each question. A cross-model knowledge index K(x) then splits failed trials into two groups. Candidate partial-retrieval failures are items where at least one model succeeds, and low-knowledge failures are items where none does. After a failure the model answers up to seven multiple-choice attribute probes, and the Partial Retrieval Score (PRS) is the fraction correct.

PRS is 0.539 for candidate failures and 0.507 for low-knowledge failures (p = 1.2e-3, Cohen's d = 0.19). The gap holds on label-stable trials. It shrinks but survives domain fixed effects, with the coefficient going from 0.174 to 0.118. Each attribute-level lead fails later checks. The pre-registered target, geographic region, does not replicate, and length bucket fails a re-draw, a forced-choice probe and a hidden-state probe. Condition labels agree across reruns only 89.5% of the time despite greedy decoding. In the cue intervention a correct cue helps both groups, with no significant group-by-cue interaction (p = 0.998). The authors narrow their claim to the aggregate PRS effect.

Critical strengths
  1. The paper is unusually honest about its own results. The authors pre-registered a target, powered a follow-up for it and report plainly that it failed. They then checked their second lead three more ways and report three nulls. They also show that condition labels agree on only 864 of 965 item-model pairs across reruns. The paper points to batch composition and 4-bit quantization as the likely cause but does not test this. Even so, it is a useful caution for work that labels whether a model knows a fact. For me this negative-results value is the main reason to read the paper.
Critical weaknesses
  1. The one claim left standing is not separated from entity prominence or from guessing based on priors. By construction, candidate failures are items some model can answer, so this group leans toward more prominent entities. Appendix I lists "partial information disappears after controlling for question difficulty and answer frequency" as falsification criterion 2. Yet the aggregate PRS is never tested against a popularity covariate or a prior-matched guessing baseline. In Table 2 the two largest per-attribute gaps are length bucket and geographic region, the two leads the paper itself shows to fail follow-up. The aggregate gap may therefore rest on the same signal, which is not target-specific.

  2. The abstract says the effect "replicates robustly", which goes beyond the evidence. The quoted p = 1.2e-3 comes from the 857-item sample itself. On the 1,310 newly drawn items in §10.1, the gap is 0.543 versus 0.530 (p = 0.130), much smaller than in the original sample. With d = 0.19 and no PRS reported for trials where retrieval succeeded, I cannot tell how much of the attainable signal this gap represents.

Questions for authors
  1. Does the aggregate gap survive a popularity covariate such as Wikidata sitelinks, and a baseline that always picks each relation's most common attribute value?
  2. What is the gap after dropping length bucket and geographic region? And after also dropping category, domain and occupation, which Table 2 shows are high and nearly identical in both groups?
  3. What PRS do trials with successful retrieval reach, and what does a question-only baseline give per attribute?
Reviewer_qDoSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether failed factual retrieval retains partial answer information. The authors test three models and divide their failed answers into two groups: cases where another model answers correctly, and cases where all three models fail. They then ask simpler questions about the missing answer, such as its geographic region or name length.The first group performs slightly better overall: 53.9% versus 50.7%. However, the region and name-length advantages do not hold up in follow-up tests, and checks inside the models find no reliable name-length information. Correct hints help both groups, but there is no clear evidence that one group benefits more.

Critical strengths
  1. The paper reports disconfirming results transparently. It identifies the synthetic-control base-rate confound, tests natural low-knowledge controls, follows attribute leads with replication and forced-choice checks, and reports label instability and hidden-state nulls.
  2. The independent attribute framework makes retrieval failure more informative than a binary answer score. Model/domain breakdowns, domain-adjusted regression, and cue controls distinguish broad accessibility, target specificity, and information supplied by a cue. These distinctions are consequential even though the remaining aggregate effect is small.
Critical weaknesses
  1. The small PRS gap may reflect differences in question difficulty and attribute composition rather than partial memory of the correct answer. The current controls do not fully separate these explanations.

  2. Follow-up tests do not consistently support the attribute-level effects, and a nonsignificant interaction does not establish that both groups benefit equally from cues.

  3. The presentation relies heavily on text and tables, with limited visual explanation. A clear overview of the experimental pipeline and plots of the main results would make the findings easier to follow and assess.

Questions for authors
  1. Does the score gap remain after controlling for question difficulty and attribute composition? How can you distinguish partial memory of the answer from guesses based on clues in the question?
  2. Does the main finding hold on an independent sample? Please clarify sample overlap, the unit of analysis, and uncertainty in the results. Does a nonsignificant difference in cue benefits justify concluding that both groups benefit equally?
  3. Can you match (T) and (I) on Wikidata popularity / pageviews / degree and still see a PRS gap?
  4. Please repeat Stage-1 labeling in bf16, batch-size 1, to see whether 89.5% agreement was a quantization artifact.
Reviewer_HLtXOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper asks the question of whether language models keep partial information about answers they fail to retrieve, like a tip-of-the-tongue state. Using Wikidata questions and three small 4-bit models, it asks multiple-choice questions about attributes of the answer after a failure. Items that some models get right score slightly higher than items no model gets right (0.539 vs 0.507). All attribute-level results fail replication, and a hidden-state probe finds nothing.

Critical strengths

The paper is very honest. It pre-registers a follow-up, reports its failed replications, and narrows its claims to what actually survives.

Critical weaknesses

The one result that holds up is small and confounded. The candidate group is defined as items that the same models sometimes answer correctly, so it’s biased toward easier, better-known entities. The authors know this and can’t rule it out. The attributes that would actually point to a specific answer, like first letter and word count, land at or below chance in both groups.

The scope is too narrow to support even a pilot claim. Three small quantized models, one prompt per item, and labels only agree 89.5% between two greedy runs of the same model.

Questions for authors
  • Can you control for entity popularity directly instead of using cross-model agreement as the split?
  • Why is first-letter accuracy below chance in both groups?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.