- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates associative-recall deficits in linear-attention language models. Comparisons of approximately 340M-parameter models trained on 15B tokens indicate that the clearest deficits relative to softmax attention occur in exact-prefix retrieval, particularly at longer distances. Synthetic MQAR experiments demonstrate substantial recall capacity and approximately linear empirical scaling with recurrent state size. Motivated by these findings, the authors introduce in-context repetition during the final stage of pretraining. This improves prefix-matching recall without increasing model state, token budget, or optimization steps, with a small deterioration in general language modeling.
Critical strengths
- The evaluation usefully distinguishes exact-prefix retrieval from broader notions of recall. The rare-token diagnostics, prompt comparisons, and context-removal and answer-substitution controls provide evidence that benchmark construction materially affects the apparent architectural gap.
- The repetition intervention is simple and produces substantial improvements on several prefix-cued benchmarks while preserving the architecture, total token budget, and number of optimization steps. The paper also reports the remaining softmax advantage and the deterioration in general language modeling, making the practical tradeoff visible.
Critical weaknesses
- The synthetic experiments do not establish that capacity is sufficient for the retrieval demands encountered during language modeling. A separately trained MQAR model allocates its representations to clean, explicitly supervised associations, with structured inputs, a designated filler token, and a curriculum at the largest scale. A language model must support many competing predictive demands within the same state. Consequently, retrieving thousands of associations in the specialized setting establishes that the architecture can implement substantial recall, but does not establish how much usable capacity is available for recall in the pretrained model. The empirical scaling results therefore do not resolve whether capacity constraints materially contribute to the observed language-model gap.
- Indeed, the repetition intervention does not isolate the proposed incentive explanation. Changing the training distribution changes the effective objective, but here it simultaneously increases copying opportunities, replaces fresh information, and shifts training toward exact-prefix continuation. The resulting gains are compatible both with learning an underdeveloped retrieval mechanism and with reallocating limited representational resources toward copying. The deterioration in general language modeling leaves this latter explanation particularly relevant. The paper establishes that training distribution affects recall, but the explanatory advance (e.g., identifying insufficient incentive as the source of the original deficit) remains insufficiently demonstrated. Although the discussion appropriately qualifies the claim, this unresolved distinction limits the central contribution.
Questions for authors
- What evidence connects the capacity measured in separately trained MQAR models to the usable recall capacity of the language-model checkpoints, where the recurrent state supports multiple competing tasks?
- Can the effects of increased retrieval supervision be separated from those of replacing fresh content?
2 comments
Dear Reviewer,
This is a kind reminder that the discussion period will end in less than 24 hours. We would appreciate it if you could take a moment to review authors' responses and provide any further comments before the discussion period closes.
Thank you for your time and effort!
We appreciate the reviewer’s distinction between demonstrating a recall operation and explaining its use during language modeling. The revision addresses this through pretrained-model copying experiments and controls that change the emphasis on retrieval without replacing training text.
1. Connecting synthetic capacity to pretrained models
The MQAR results demonstrate attainable storage and retrieval, not spare capacity in a language-model checkpoint. Our additional fixed-state sweeps vary width, parameter count, and head layout using LA. Retrieval changes comparatively little across these choices and remains primarily governed by state size (§4.2, Figure 4). This clarifies the synthetic result without resolving competition among language-modeling functions.
The new pretrained-model experiment provides a closer connection (§5.1, Figure 7). When asked to reproduce a 4,096-token natural-text passage, DN performs near its text-only baseline of 50%, despite having the passage in context. Copy fine-tuning raises teacher-forced accuracy to 99.8% with the same state size. The model can therefore acquire a retrieval behavior that its original training did not reliably develop.
This is not evidence that the relevant resources were previously unused. Dedicated copying reduces text-only accuracy from 49.4% to 22.4%, and resource reallocation remains a plausible mechanism. We have made that distinction explicit in §§6 and 8. The contribution is evidence of substantial trainable recall within the existing architecture, rather than exclusion of capacity constraints from the explanation.
2. Separating recall supervision from fresh-content replacement
Loss reweighting supplies the direct control for content replacement. With the training text unchanged, increasing the loss weight of AR-hit tokens by a factor of five improves extraction, QA, and prefix accuracy. Larger gains come from Thresholded LongCE (TLCE), our modification of LongCE (Fang et al., 2025), which upweights token losses when longer context improves the target token’s log probability by more than one nat. At 340M, TLCE increases these accuracies from 21.1%, 30.1%, and 43.6% to 29.2%, 35.2%, and 51.9%, respectively, with a held-out LM loss increase of 0.0333 nats (§6.2; Appendix H.1, Table 10). Thus, neither recall improvement nor an accompanying LM cost requires displacement of fresh content.
The duration experiments address whether the cost grows with cumulative replacement. At 61.7M, inserting repeated sentence spans throughout training produces much greater recall gains than restricting this intervention to the final cooldown, at nearly unchanged LM cost (Appendix H.2, Table 12).
The shares are approximately matched, and the total training budget is unchanged. Despite replacing fresh text over a much larger fraction of training, the full-training runs more than double the prefix accuracy gains without a proportional increase in LM cost.
At the main model scale, applying repetition and TLCE throughout KDA training raises extraction accuracy from 39.1% to 48.9% relative to the combined cooldown intervention, approaching standard-trained softmax at 52.1%. The mean LAMBADA/WikiText perplexity penalty remains comparable at 1.7 versus 1.9 (§7, Figure 9; Appendix H.2).
These controls establish that replacement is unnecessary for the gains and that cumulative displacement does not straightforwardly determine the observed penalty. They do not fully decompose repetition’s effects or distinguish learning a retrieval mechanism from reallocating state resources. We believe they address the fresh-content concern while giving the paper’s explanatory claim a clearer, appropriately limited scope.