The Knowledge Is Already There: Geometric Interventions for Factual Recall in LLMs

Factual retrieval failures in large language models are often assumed to reflect missing knowledge. We investigate an alternative mode: representational collapse, in which questions requiring different answers occupy overlapping regions of the hidden-state space. We measure this by a question’s crowdedness, the number of other questions with highly similar representations. Across three sizes of Qwen2.5, the more crowded half of TriviaQA questions is answered correctly 22–25 percentage points less often than the rest. We propose Contrastive Representation Adjustment for Factuality (CRAFT), which reshapes this geometry through a contrastive objective over question paraphrases and negatives, without any answer supervision. Evaluated on TriviaQA, NQ-Open, and HotpotQA across three model families, CRAFT gives the best exact-match accuracy of all methods on every benchmark for all Qwen2.5 sizes, raising Qwen2.5-7B accuracy on TriviaQA from 38.4% to 58.2%. Finally, CRAFT largely removes representational collapse, consistent with many retrieval failures reflecting knowledge that is present in the model but hard to reach