- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether the density of relevant information in context affects LLM reasoning performance. The authors introduce SmolBench, consisting of synthetic deduction and induction tasks where the same task-relevant content is represented either compactly or through longer, lower-density representations. Across 16 models, the paper reports that high-density representations generally achieve substantially higher accuracy than low-density representations, even when irrelevant information is appended to match context length. The low-density condition also produces much longer reasoning traces, with an average increase of approximately 4.98×. The authors interpret these results as evidence that models attend less effectively to low-density relevant information and consequently follow longer or invalid reasoning chains. While the empirical pattern is interesting and observed across many models, the current benchmark design does not cleanly isolate information density from changes in task representation, search difficulty, and algorithmic complexity.
Critical strengths
-
The paper identifies an important and underexplored distinction between irrelevant context and low-density relevant context. The observation that relevant information can be harmful when represented inefficiently is practically important for retrieval, agentic systems, and long-context reasoning.
-
The experiments cover 16 models from multiple model families and parameter scales, and include both deductive and inductive synthetic tasks. The consistent direction of many results, together with the reasoning-length and proof- trace analyses, provides meaningful empirical evidence that the phenomenon is not confined to a single model.
Critical weaknesses
-
The benchmark does not cleanly isolate information density from representation-induced reasoning difficulty. In SmolBench-Deduction, the low-density condition adds derivation trees that create substantially more intermediate statements and a larger search space, even though they are called relevant. In SmolBench-Induction, the two conditions require qualitatively different algorithms: deriving counts from compact rules versus scanning or reconstructing a long enumeration. Therefore, the observed accuracy gap may be explained by search complexity, sequence length, branching, or algorithmic mismatch rather than by lower relevance per token.
-
The proposed attention-based mechanism is not directly validated. The paper infers that low-density information receives less attention and causes longer, worse reasoning chains, but it does not report attention measurements, attention-mass comparisons, causal attention interventions, or controlled experiments that keep the underlying reasoning procedure fixed. Longer reasoning traces are compatible with the proposed explanation, but they do not establish it; they may simply reflect that the low-density representation requires a harder computation.
Questions for authors
-
Can the authors construct controls in which the underlying reasoning algorithm and search space are held fixed while only the token-level representation density changes? For example, can the same proof graph be presented with different amounts of verbal redundancy or equivalent compression, while preserving the same valid intermediate states and branching structure?
-
For SmolBench-Induction, how do the authors rule out the possibility that the low-density condition is simply a harder algorithmic task involving enumeration, whereas the high-density condition provides an explicit shortcut through the generating rules?
-
Can the authors provide direct evidence for the attention-based explanation, such as attention mass assigned to the relevant statements, causal attention masking/patching experiments, or comparisons under a fixed reasoning-token budget?
-
Please clarify the averaging procedure behind the reported overall degradation numbers. The paper reports several different averages, including approximately 30.1%, 30.2%, and 12.1%, and it would be helpful to distinguish clearly between high-versus-low-density differences and irrelevant-versus-low-density differences.