acceptodds
Under review as a conference paper at ICLR 2027

Less is More: Higher Information Density Increases Performance

Abstract

In order for LLMs to target larger and more complex tasks, they are given increasing amounts of information in context meant to help them solve those tasks. Prior work has shown that giving LLMs irrelevant information, such as random facts or whitespace, degrades LLM performance on tasks because attention is diluted by attending to irrelevant information. However, complex tasks often involve the LLM ingesting information that could be used to solve the task (i.e., relevant information) of varying degrees of relevance, rather than purely irrelevant information. In this paper, we investigate the role of relevant information density, or the amount of relevant information in context given to an LLM. We find that denser representations of relevant information perform better, even when controlling for context length by appending irrelevant information. We show this is consistent with LLMs attending to more than the minimum possible information they need, causing distraction, which we show with higher reasoning token lengths. We construct benchmark tasks which allow us to precisely control the amount and density of relevant information in context and find that, across 16 models, performance degrades on average 30.1 percentage points on all tasks due to the difference in information density, despite the model seeing the same relevant information. This degradation is accompanied by a increase in reasoning length. Our results show that managing context to maximize information density can improve model performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
context management, information representation, long context

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DPFqOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether the density of relevant information in context affects LLM reasoning performance. The authors introduce SmolBench, consisting of synthetic deduction and induction tasks where the same task-relevant content is represented either compactly or through longer, lower-density representations. Across 16 models, the paper reports that high-density representations generally achieve substantially higher accuracy than low-density representations, even when irrelevant information is appended to match context length. The low-density condition also produces much longer reasoning traces, with an average increase of approximately 4.98×. The authors interpret these results as evidence that models attend less effectively to low-density relevant information and consequently follow longer or invalid reasoning chains. While the empirical pattern is interesting and observed across many models, the current benchmark design does not cleanly isolate information density from changes in task representation, search difficulty, and algorithmic complexity.

Critical strengths
  1. The paper identifies an important and underexplored distinction between irrelevant context and low-density relevant context. The observation that relevant information can be harmful when represented inefficiently is practically important for retrieval, agentic systems, and long-context reasoning.

  2. The experiments cover 16 models from multiple model families and parameter scales, and include both deductive and inductive synthetic tasks. The consistent direction of many results, together with the reasoning-length and proof- trace analyses, provides meaningful empirical evidence that the phenomenon is not confined to a single model.

Critical weaknesses
  1. The benchmark does not cleanly isolate information density from representation-induced reasoning difficulty. In SmolBench-Deduction, the low-density condition adds derivation trees that create substantially more intermediate statements and a larger search space, even though they are called relevant. In SmolBench-Induction, the two conditions require qualitatively different algorithms: deriving counts from compact rules versus scanning or reconstructing a long enumeration. Therefore, the observed accuracy gap may be explained by search complexity, sequence length, branching, or algorithmic mismatch rather than by lower relevance per token.

  2. The proposed attention-based mechanism is not directly validated. The paper infers that low-density information receives less attention and causes longer, worse reasoning chains, but it does not report attention measurements, attention-mass comparisons, causal attention interventions, or controlled experiments that keep the underlying reasoning procedure fixed. Longer reasoning traces are compatible with the proposed explanation, but they do not establish it; they may simply reflect that the low-density representation requires a harder computation.

Questions for authors
  1. Can the authors construct controls in which the underlying reasoning algorithm and search space are held fixed while only the token-level representation density changes? For example, can the same proof graph be presented with different amounts of verbal redundancy or equivalent compression, while preserving the same valid intermediate states and branching structure?

  2. For SmolBench-Induction, how do the authors rule out the possibility that the low-density condition is simply a harder algorithmic task involving enumeration, whereas the high-density condition provides an explicit shortcut through the generating rules?

  3. Can the authors provide direct evidence for the attention-based explanation, such as attention mass assigned to the relevant statements, causal attention masking/patching experiments, or comparisons under a fixed reasoning-token budget?

  4. Please clarify the averaging procedure behind the reported overall degradation numbers. The paper reports several different averages, including approximately 30.1%, 30.2%, and 12.1%, and it would be helpful to distinguish clearly between high-versus-low-density differences and irrelevant-versus-low-density differences.

Reviewer_PFUPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how representations of task-relevant information affect LLM reasoning. The authors introduce deductive proof and periodic-sequence counting tasks, evaluating 16 models under high-density, low-density, and length-matched high-density-plus-irrelevant conditions. Low-density representations generally lead to lower accuracy and longer reasoning.

Critical strengths
  1. The paper examines information that can contribute to valid reasoning but is not necessary, helping explain why adding correct, relevant material may still hurt performance.

  2. The low-density condition preserves the original lemmas and short proof path, preventing the performance drop from being attributed simply to removing the shorter solution.

  3. Cross-model experiments, length-matched controls, and automated proof-path analysis provide complementary evidence for the observed effects.

Critical weaknesses
  1. The induction experiment confounds information density with computational burden. The high-density condition directly provides generating rules from which symbol counts can be calculated, whereas the low-density condition requires counting an enumerated sequence or recovering its rules. Although both conditions yield the same answer, they provide different computational routes. The performance gap may therefore reflect differences in computational difficulty rather than information density, weakening this experiment’s support for the central claim.

  2. The deduction experiment does not establish a general information-density effect. Preserving the original short proof while adding derivation trees is a useful design. However, the additional material also introduces more candidate rules and alternative proof paths. The experiment does not disentangle representation length, rule count, and competition among proof paths. Length-matched irrelevant padding controls token count alone. The results therefore support the narrower finding that adding redundant, valid proofs can hurt performance, but do not yet establish a general information-density effect.

  3. The mechanistic explanation exceeds the available evidence. The authors attribute performance degradation to difficulty attending to the right information, based on increased reasoning length and changes in proof paths. These behavioral observations are also compatible with increased search burden or suboptimal strategy selection. The paper neither directly measures nor intervenes on the proposed attention mechanism, and the standard attention equations do not resolve this evidential gap. The causal claim that attention dilution leads to errors should therefore be presented as a hypothesis requiring further validation.

Questions for authors
  1. Could the authors separately control the number of additional rules, the number of alternative proofs, and derivation depth? What evidence distinguishes an information-density effect from search interference caused by additional candidate paths?

  2. Could targeted interventions distinguish attention allocation from proof-path selection—for example, constraining models to use the original lemmas, blocking alternative paths, or intervening on relevant attention connections? Otherwise, could the causal claims be qualified and the main conclusions restricted to the observed behavioral findings?

Reviewer_MydROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Previous work have shown that passing more information to LLMs have risk, and authors argue that how relevant information affects LLM's behavior is underexplored. This paper introduces SmolBench, a synthetic benchmark that controls the relevance and density of information. The authors conclude that denser representations generally result in higher accuracy and that context length alone does not explain the observed performance differences.

Critical strengths
  1. The deduction construction preserves an available shortest proof while adding optional valid derivations. This means the performance degradation cannot simply be attributed to an increase in the minimum number of proof steps required.

  2. The deduction results show a consistent direction across the evaluated models.

Critical weaknesses
  1. The experiments support narrower conclusions than the proposed explanation. In deduction, comparing added valid derivation trees with lorem ipsum does not distinguish effects specific to usable alternative proofs from distraction caused by additional task-like logical structure. The fixed unusable alternatives already present in all conditions do not provide a matched control for this addition. The reasoning-length and proof-route analysis are consistent with less effective route selection, but do not necessarily establish an attention-based mechanism. In induction, supplying generating rules rather than an enumeration also changes computational accessibility. These limitations reduce my confidence in the broader claim that an independent information-density effect has been isolated.

  2. The incremental contribution needs clearer positioning relative to close prior work. GSM-Infinite (https://arxiv.org/abs/2502.05252v1), the relevant Physics of Language Models studies (https://physics.allen-zhu.com/part-2-grade-school-math/part-2-1), and Intrinsic Entropy of Context Length Scaling(https://openreview.net/forum?id=vnipyA8c9V) already investigate related aspects of context, structured reasoning, and information representation. The most distinctive contribution here appears to be the addition of optional valid derivations while retaining the original shortest proof. The paper should explain more precisely what new insight this construction provides beyond those precedents. This uncertainty limits the conceptual significance I assign to the work.

Questions for authors
  1. What existing evidence distinguishes interference specific to usable alternative proofs from equally structured but unusable additions, and supports the attention-based interpretation?

  2. What specific insight does the shortest-proof-preserving construction establish beyond the effects examined in the related work above?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.