Less is More: Higher Information Density Increases Performance

In order for LLMs to target larger and more complex tasks, they are given increasing amounts of information in context meant to help them solve those tasks. Prior work has shown that giving LLMs irrelevant information, such as random facts or whitespace, degrades LLM performance on tasks because attention is diluted by attending to irrelevant information. However, complex tasks often involve the LLM ingesting information that could be used to solve the task (i.e., relevant information) of varying degrees of relevance, rather than purely irrelevant information. In this paper, we investigate the role of relevant information density, or the amount of relevant information in context given to an LLM. We find that denser representations of relevant information perform better, even when controlling for context length by appending irrelevant information. We show this is consistent with LLMs attending to more than the minimum possible information they need, causing distraction, which we show with higher reasoning token lengths. We construct benchmark tasks which allow us to precisely control the amount and density of relevant information in context and find that, across 16 models, performance degrades on average 30.1 percentage points on all tasks due to the difference in information density, despite the model seeing the same relevant information. This degradation is accompanied by a $4.98\times$ increase in reasoning length. Our results show that managing context to maximize information density can improve model performance.