acceptodds
Under review as a conference paper at ICLR 2027

Linear Attention lacks Incentive for Associative Recall, not Capacity

Abstract

Linear-attention models exhibit disproportionate deficits in prefix-matching associative recall, particularly at long retrieval distances, even when their overall language-modeling loss matches or improves on softmax attention. Motivated by the performance gap, we revisit both real and synthetic recall tasks with different linear attention architectures, narrowly defining the recall-gap at an academic LLM scale. Consistent with prior work, we observe that this architecture-dependent gap is concentrated on precise predictions requiring prefix-matching associative recall. To investigate this gap, we use MQAR and find a shared empirical capacity frontier across the tested linear-attention architectures. Extrapolating this scaling, we find that a model matching a single layer of a trained LLM, can learn to retrieve thousands of synthetic associations across long distances. These results suggest that capacity alone does not explain the natural-language recall gap at this scale. We instead hypothesize that standard pretraining provides insufficient supervision for robust associative retrieval. A simple in-context repetition scheme improves recall-intensive benchmark performance and narrows the long-distance prefix-matching gap without changing the architecture, token budget, or number of optimization steps, at a slight cost to general language modeling. Targeted changes to the training distribution can thus elicit substantial but underutilized retrieval capacity in linear-attention models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Linear attention, associative recall, in-context retrieval, memory capacity, language modelling, recurrent architectures, Delta Net

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NcagOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates associative-recall deficits in linear-attention language models. Comparisons of approximately 340M-parameter models trained on 15B tokens indicate that the clearest deficits relative to softmax attention occur in exact-prefix retrieval, particularly at longer distances. Synthetic MQAR experiments demonstrate substantial recall capacity and approximately linear empirical scaling with recurrent state size. Motivated by these findings, the authors introduce in-context repetition during the final stage of pretraining. This improves prefix-matching recall without increasing model state, token budget, or optimization steps, with a small deterioration in general language modeling.

Critical strengths
  1. The evaluation usefully distinguishes exact-prefix retrieval from broader notions of recall. The rare-token diagnostics, prompt comparisons, and context-removal and answer-substitution controls provide evidence that benchmark construction materially affects the apparent architectural gap.
  2. The repetition intervention is simple and produces substantial improvements on several prefix-cued benchmarks while preserving the architecture, total token budget, and number of optimization steps. The paper also reports the remaining softmax advantage and the deterioration in general language modeling, making the practical tradeoff visible.
Critical weaknesses
  1. The synthetic experiments do not establish that capacity is sufficient for the retrieval demands encountered during language modeling. A separately trained MQAR model allocates its representations to clean, explicitly supervised associations, with structured inputs, a designated filler token, and a curriculum at the largest scale. A language model must support many competing predictive demands within the same state. Consequently, retrieving thousands of associations in the specialized setting establishes that the architecture can implement substantial recall, but does not establish how much usable capacity is available for recall in the pretrained model. The empirical scaling results therefore do not resolve whether capacity constraints materially contribute to the observed language-model gap.
  2. Indeed, the repetition intervention does not isolate the proposed incentive explanation. Changing the training distribution changes the effective objective, but here it simultaneously increases copying opportunities, replaces fresh information, and shifts training toward exact-prefix continuation. The resulting gains are compatible both with learning an underdeveloped retrieval mechanism and with reallocating limited representational resources toward copying. The deterioration in general language modeling leaves this latter explanation particularly relevant. The paper establishes that training distribution affects recall, but the explanatory advance (e.g., identifying insufficient incentive as the source of the original deficit) remains insufficiently demonstrated. Although the discussion appropriately qualifies the claim, this unresolved distinction limits the central contribution.
Questions for authors
  1. What evidence connects the capacity measured in separately trained MQAR models to the usable recall capacity of the language-model checkpoints, where the recurrent state supports multiple competing tasks?
  2. Can the effects of increased retrieval supervision be separated from those of replacing fresh content?
2 comments

We appreciate the reviewer’s distinction between demonstrating a recall operation and explaining its use during language modeling. The revision addresses this through pretrained-model copying experiments and controls that change the emphasis on retrieval without replacing training text.

1. Connecting synthetic capacity to pretrained models

The MQAR results demonstrate attainable storage and retrieval, not spare capacity in a language-model checkpoint. Our additional fixed-state sweeps vary width, parameter count, and head layout using LA. Retrieval changes comparatively little across these choices and remains primarily governed by state size (§4.2, Figure 4). This clarifies the synthetic result without resolving competition among language-modeling functions.

The new pretrained-model experiment provides a closer connection (§5.1, Figure 7). When asked to reproduce a 4,096-token natural-text passage, DN performs near its text-only baseline of 50%, despite having the passage in context. Copy fine-tuning raises teacher-forced accuracy to 99.8% with the same state size. The model can therefore acquire a retrieval behavior that its original training did not reliably develop.

This is not evidence that the relevant resources were previously unused. Dedicated copying reduces text-only accuracy from 49.4% to 22.4%, and resource reallocation remains a plausible mechanism. We have made that distinction explicit in §§6 and 8. The contribution is evidence of substantial trainable recall within the existing architecture, rather than exclusion of capacity constraints from the explanation.

2. Separating recall supervision from fresh-content replacement

Loss reweighting supplies the direct control for content replacement. With the training text unchanged, increasing the loss weight of AR-hit tokens by a factor of five improves extraction, QA, and prefix accuracy. Larger gains come from Thresholded LongCE (TLCE), our modification of LongCE (Fang et al., 2025), which upweights token losses when longer context improves the target token’s log probability by more than one nat. At 340M, TLCE increases these accuracies from 21.1%, 30.1%, and 43.6% to 29.2%, 35.2%, and 51.9%, respectively, with a held-out LM loss increase of 0.0333 nats (§6.2; Appendix H.1, Table 10). Thus, neither recall improvement nor an accompanying LM cost requires displacement of fresh content.

The duration experiments address whether the cost grows with cumulative replacement. At 61.7M, inserting repeated sentence spans throughout training produces much greater recall gains than restricting this intervention to the final cooldown, at nearly unchanged LM cost (Appendix H.2, Table 12).

Repetition sharePrefix accuracy gain, cooldown / throughoutLM loss increase, cooldown / throughout
Approximately 10%+8.0 / +21.4 points+0.029 / +0.029 nats
Approximately 20%+11.3 / +25.9 points+0.057 / +0.060 nats

The shares are approximately matched, and the total training budget is unchanged. Despite replacing fresh text over a much larger fraction of training, the full-training runs more than double the prefix accuracy gains without a proportional increase in LM cost.

At the main model scale, applying repetition and TLCE throughout KDA training raises extraction accuracy from 39.1% to 48.9% relative to the combined cooldown intervention, approaching standard-trained softmax at 52.1%. The mean LAMBADA/WikiText perplexity penalty remains comparable at 1.7 versus 1.9 (§7, Figure 9; Appendix H.2).

These controls establish that replacement is unnecessary for the gains and that cumulative displacement does not straightforwardly determine the observed penalty. They do not fully decompose repetition’s effects or distinguish learning a retrieval mechanism from reallocating state resources. We believe they address the fresh-content concern while giving the paper’s explanatory claim a clearer, appropriately limited scope.

Dear Reviewer,

This is a kind reminder that the discussion period will end in less than 24 hours. We would appreciate it if you could take a moment to review authors' responses and provide any further comments before the discussion period closes.

Thank you for your time and effort!

Reviewer_F7YPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper first investigates the recall gap between linear attention and softmax attention through synthetic MQAR and real-world recall benchmarks, showing that the gap is primarily concentrated on prefix-matching associative retrieval. The authors further observe that MQAR capacity scales approximately linearly with recurrent state size, and that a model with LLM-scale state capacity can retrieve thousands of KV pairs. Based on this, they hypothesize that the main limitation of linear attention is insufficient training incentive for retrieval rather than limited state capacity. To mitigate this, the authors propose an in-context repetition scheme during the last stage of training, which substantially improves retrieval performance at a slight cost to language modeling performance.

Critical strengths
  1. The paper is well-written, with a clear and easy-to-understand presentation of its methodology.
  2. The paper follows a coherent narrative, with each step of the argument supported by relevant experimental evidence.
Critical weaknesses
  1. Insufficiently rigorous justification of the central claim. The paper's motivation relies heavily on synthetic MQAR experiments, leaving a substantial gap between the experimental evidence and the conclusions about real language modeling. Moreover, the claim that the primary bottleneck is training incentive rather than state capacity is not sufficiently justified. See Questions (1)–(2).
  2. Insufficient experimental validation. The paper would benefit from more realistic and controlled ablations, a clearer explanation of the mixed effects, and further investigation of scaling behavior and the retrieval–language modeling trade-off. See Questions (3)–(5).
Questions for authors

1. Insufficient evidence that the recall gap is primarily due to training incentives rather than state capacity. The paper draws this conclusion mainly from synthetic MQAR experiments and extrapolates the observed capacity scaling to pretrained language models. However, MQAR involves simple, fixed-pattern associative recall, whereas real language modeling requires the same recurrent states to handle more complex and noisy retrieval (e.g., M-NIAH), language understanding, and context modeling. Therefore, the effective recall capacity per state bit may be substantially lower than the MQAR estimate. Moreover, the LLM-scale MQAR experiment uses a ramping schedule designed to improve large-scale MQAR performance, further limiting the generality of the results. I believe these experiments are insufficient to support the claim that retrieval gaps are not primarily caused by limited state capacity.

2. Does improved retrieval through repetition training necessarily indicate insufficient training incentives? Increasing the proportion of task-relevant training samples is a general strategy for improving performance on the corresponding distribution. Therefore, better retrieval performance after repetition training does not directly establish that the primary bottleneck is insufficient training incentives rather than limited state capacity or other factors. Even under capacity constraints, additional retrieval-focused training may help models utilize their limited capacity more effectively. Thus, the observed improvements do not sufficiently distinguish between these explanations.

3. More controlled ablations are needed. A single repetition-training experiment is insufficient to support the paper's central claim about the bottleneck of linear attention (LA). I suggest at least two additional experiments: (1) varying the repetition ratio to examine how retrieval performance scales with the amount of repetition; and (2) increasing the loss weight of AR-hit tokens to test whether similar improvements can be achieved by modifying the training objective rather than the data distribution, since both approaches can strengthen the incentive for retrieval.

4. Limited and inconsistent improvements on NIAH. The NIAH results show mixed effects of repetition training on LA, with improvements mainly appearing under specific evaluation settings. Could this be because the standard prompts do not align well with the repetition patterns used during training, while prefix-based prompts are more closely matched? In contrast, softmax attention appears to benefit more consistently from repetition training. These results raise questions about the underlying mechanism and generalizability of repetition training + LA.

5. Unclear scaling behavior and retrieval–language modeling trade-off. Increasing the proportion of task-specific training data can improve in-domain performance at the cost of general language modeling, consistent with the observed retrieval gains and PPL degradation. However, the paper only evaluates this approach at a single model scale and token budget. It remains unclear how the retrieval gains and PPL degradation change with model and data scaling, and whether this trade-off remains worthwhile at larger scales.

4 comments

We thank the reviewer for the assessment and suggestions.

1. Justification of the capacity claim

To address your concern about MQAR’s synthetic key–value structure, we added copying experiments progressing from random sequences to natural text and pretrained models (§§5–5.1). Copying tests passage retrieval and directly motivates repetition.

Copying follows similar state-size scaling to MQAR, with natural text easier than random tokens (Table 2; Figure 6). On 4,096-token passages, pretrained DN performs near its 50% text-only baseline despite having the passage in context. Copy fine-tuning raises teacher-forced accuracy to 99.8% without enlarging the state (Figure 7). However, text-only accuracy falls from 49.4% to 22.4%, so this establishes attainable retrieval, not its coexistence with all other capabilities unchanged.

2. What do repetition gains establish?

We agree that repetition gains alone do not identify insufficient incentives as the primary bottleneck. Our interpretation does not require unused state capacity: reallocating finite resources toward retrieval is a possible mechanism through which training incentives affect recall, rather than an alternative to that explanation.

We distinguish this contribution of training from an exclusive explanation of the deficit (§§6 and 8). Recall can improve substantially without enlarging the state, while the contribution of competing demands remains unresolved. Softmax also exhibits recall gains with LM costs (Table 9), so this tradeoff alone does not identify competition for recurrent-state memory.

3. Controlled repetition and loss-reweighting ablations

As requested, the KDA 340M ablations fix the starting checkpoint, data order, optimization schedule, and step budget during cooldown (Appendix H.1, Table 10). Sentence repetition inserts spans averaging 64 tokens at later sentence boundaries; rates denote copied-token shares.

Following your suggestion, we test loss reweighting by increasing the loss weight of AR-hit tokens by a factor of five and additionally evaluate LongCE, which upweights predictions benefiting from longer context. Our Thresholded LongCE (TLCE) applies this weighting only when the full-context log-probability advantage exceeds one nat (§6.2).

InterventionExtraction ↑Prefix ↑Δ LM loss ↓
Baseline21.143.6—
Repeat 5%29.556.2+0.0071
Repeat 10%31.258.4+0.0123
Repeat 22%32.161.2+0.0241
Repeat 58%29.760.6+0.0686
Fivefold loss weight on AR-hit tokens23.649.5+0.0163
LongCE29.352.3+0.0867
TLCE29.251.9+0.0333
TLCE + Repeat 10%39.166.1+0.0538

Extraction averages SWDE/FDA accuracy; Prefix averages all six exact-prefix tasks (both in %). Δ LM loss is the held-out FineWeb-Edu increase in nats.

Higher repetition rates yield diminishing gains at greater LM cost, supporting 10% as a compromise. Sentence repetition also outperforms full-sequence repetition at comparable cost. Loss reweighting improves recall without replacing fresh text: direct AR-hit weighting yields smaller gains, while LongCE and TLCE provide larger improvements that complement repetition, with TLCE incurring a lower LM cost. Combined training also closes the long-distance AR-hit loss gap to standard-trained softmax across all four linear mixers, although softmax retains an advantage under matched interventions (Appendix I).

4. Mixed NIAH results

Your concern about prompt alignment is supported by the expanded results (§7.1; Appendix J, Tables 14–15). Repetition benefits exact-prefix prompts most; on standard NIAH-2 at 4K, KDA improves while GDN, DN, and GLA decline. Some aggregate declines reflect single-model failures: combined training lowers GDN’s colon-prompt NIAH-2 accuracy from 72% to 4%, outweighing improvements in the other three architectures.

Scores also swing with length under the same prompt: KDA trained throughout with repetition and TLCE scores 99%, 40%, and 97% on standard NIAH-2 at 2K, 4K, and 8K. TLCE transfers more broadly on word retrieval, but neither intervention is uniformly reliable. Teacher-forced results show more consistent improvements, indicating better token prediction when the preceding answer tokens are supplied, even where generating the complete answer remains unreliable.

5. Scaling and the retrieval–LM tradeoff

The 61.7M experiments broadly reproduce the trends at 340M (Appendix H.2, Table 11). We also compare interventions applied throughout training with those restricted to the final cooldown, holding the total training budget fixed. Longer exposure produces substantially greater recall gains at comparable LM cost, for repetition at 61.7M and repetition with TLCE at 340M (Table 12; §7, Figure 9). Thus, recall continues to improve with longer intervention exposure without requiring more training tokens, and the observed tradeoff depends on training design.

Dear Reviewer,

This is a kind reminder that the discussion period will end in less than 24 hours. We would appreciate it if you could take a moment to review authors' responses and provide any further comments before the discussion period closes.

Thank you for your time and effort!

I appreciate the authors' detailed responses, especially their acknowledgment of the limitations, which substantially improves the paper's rigor and credibility.

However, I still believe the paper mainly demonstrates a trade-off between retrieval and general language modeling: strengthening retrieval supervision improves recall, but at the cost of language modeling performance. This observation is largely intuitive, and the paper does not provide an effective solution to this trade-off. Moreover, the authors acknowledge that retrieval improvements remain inconsistent across evaluation settings, particularly on NIAH. These limitations restrict the paper's practical contributions. Therefore, I will maintain my original score.

I thank the authors again for their thorough responses and additional experiments.

Thank you for taking another look at the paper and for your thoughtful feedback. We agree that the recall gains come at some cost to language modeling. We see these relatively simple methods as a step toward a better trade-off to recover the strong recall of softmax under the architecture constraints, with room for further improvement.

Weak recall has been a central motivation for softmax attention layers rather than adopting full linear models and considering that what we think matters in practice is that training alone can bring fully linear models closer to softmax on recall tasks while preserving the efficient scaling with sequence length and low inference cost that make them attractive for long-context applications. The fact that it costs in LM performance is true, but since they still perform competitively, with the best KDA model actually still beating the baseline softmax on WikiText and Lambada while coming close to matching the baseline on recall tasks. Applying sentence repetition and TLCE throughout KDA training brings its extraction accuracy close to standard-trained softmax, with about the same measured perplexity increase as applying them only during cooldown. Even during cooldown, the combined intervention improves extraction by 18.5 percentage points on average across the linear models, roughly twice the spread between the strongest and weakest of those models under standard training (Table 9). Given the architectural work devoted to improving recall, including Gated DeltaNet-2, we think the size of this gain makes the training objective worth considering alongside the choice of mixer.

Given the inconsistent NIAH results, we carried out a supplemental analysis of the logged generations. We focused on standard-prompt NIAH-2, where full-training KDA scores 99% at 2K, falls to 40% at 4K, and recovers to 97% at 8K. At 4K, the model repeatedly writes _____ where the number should go. Every example at that length uses the same essay text, which contains five underscore separators, the last about 200 tokens before the question. Their proximity offers a plausible explanation for why the model so often continues the prompt as a fill-in-the-blank exercise, while it still seems to be able to perform accurate retrieval at length. This illustrates how sensitive NIAH can be to the particular haystack and cue. The 40% score captures a real failure to answer the standard question, but the table alone makes the drop look more like a loss of long-distance recall than the generations suggest. Across tasks and lengths, full-training KDA still outperforms both baseline and cooldown-only KDA on the standard NIAH prompt (Table 3), and the combined intervention also improves accuracy on the original QA prompts (Table 9), so these harmful effects appear to be contained to this setting. To test how the haystack itself contributes, we try swapping it to using a set of unseen fineweb-edu sequences. As the table below shows changing the hay stack itself to these fineweb sequences more consistently shows the improvement of each intervention. This is using the standard prompt so without the matching prefix.

MixerRunFineWeb 2KFineWeb 4KFineWeb 8KEssay 2KEssay 4K
SoftmaxBase0.9980.878–1.0000.982
SoftmaxLSD+Splice0.9980.996–1.0000.994
KDABase0.5240.208–0.8080.546
KDASplice0.6440.366–0.6440.580
KDALSD+Splice (cooldown)0.8380.622–0.9640.512
KDAFull0.9680.8920.6100.9880.404

Both the splice and combined results improve clearly in this setting over the baseline, and the intervention with both over the full run closely matches the baseline Softmax in this task as well, when the above local failure mode is avoided. We are currently rescoring the rest of the results in this setting and will update the manuscript with these additions as soon as possible.

Thank you again for the careful review and we hope these answers help resolve your concerns.

Reviewer_a2xHOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper examines whether the recall gap between linear and softmax attention reflects limited state capacity or insufficient training incentives. It localizes the deficit to precise recall with matching prefixes, especially over long distances, and demonstrates substantial recall capacity in dedicated synthetic experiments. Motivated by the scarcity of explicit recall supervision in pretraining, the authors introduce repetitions into the training text. This improves recall without changing the architecture or training token budget, supporting the hypothesis that the training distribution contributes to the gap.

Critical strengths
  1. The paper provides a coherent empirical investigation connecting the diagnosis of recall failures, synthetic capacity measurements, and an intervention in language model training. Rather than treating recall as a single capability, it identifies precise prefix matching as a particular weakness and provides evidence that training choices can improve this capability at a fixed state size.

  2. The experiments and analyses are extensive, covering multiple modern linear attention architectures, natural language and synthetic tasks, state sizes, retrieval distances, and prompt conditions. Context removal and answer substitution strengthen the interpretation of the retrieval results. Improvements on held-out natural text complement the benchmark gains, while the reported language modeling costs and prompt sensitivity clarify their scope.

Critical weaknesses
  1. The authors acknowledge capacity constraints and competition with other language modeling demands, and appropriately present their explanation as a hypothesis. However, capacity demonstrated under dedicated MQAR training does not establish how much is available for recall during language modeling. Repetition improves recall but slightly worsens general language modeling, a pattern also compatible with reallocating limited state resources toward precise recall. Although this does not establish a capacity bottleneck, it leaves the relative contributions of stronger supervision and capacity allocation unresolved.
Questions for authors

What additional evidence could distinguish insufficient recall supervision from limitations in the state capacity available for recall during language modeling? The synthetic experiments demonstrate substantial capacity under dedicated recall training, but it remains unclear how much of that capacity is available when the model must also support other language modeling demands.

2 comments

We thank the reviewer for the thoughtful assessment and for recognizing the connection between the recall diagnostics, synthetic experiments, and training intervention.

We agree that dedicated recall training does not establish how much capacity remains available alongside other language-modeling demands. Our intended distinction is between the capacity to represent effective recall and how training develops that capability. The added copying experiments extend the evidence beyond MQAR to natural text and pretrained models (§§5–5.1). If stronger supervision improves recall by reallocating resources from other predictive functions, this is compatible with the incentive hypothesis.

The expanded experiments also show that the measured recall–LM tradeoff depends on training design. We added loss-reweighting experiments based on LongCE (Fang et al., 2025), which gives greater weight to token predictions that benefit from longer context. Our Thresholded LongCE (TLCE) variant applies this weighting only when the full-context log-probability advantage exceeds one nat, leaving other raw weights at one. TLCE retains recall performance close to LongCE while reducing the held-out LM loss increase from 0.0867 to 0.0333 nats (Appendix H.1, Table 10). Longer exposure can yield substantially greater recall at comparable cost. At 61.7M, sentence repetition near 10% improves prefix accuracy by 21.4 points when applied throughout training, versus 8.0 points during cooldown, with the same measured LM penalty of 0.029 nats (Appendix H.2, Table 12). Applying sentence repetition and TLCE together throughout KDA training, rather than only during cooldown, provides complementary evidence at 340M (§7, Figure 9). These findings suggest that the original tradeoff was not an established limit imposed by state size.

Moreover, softmax also exhibits recall gains accompanied by LM costs (Appendix G, Table 9), so that pattern alone does not identify competition for a fixed recurrent state as its cause.

We have clarified in §§6 and 8 that our claim concerns attainable recall capability, not otherwise unused memory. The results demonstrate that training can develop substantially better recall within the existing architecture; they do not determine how much of that improvement comes from reallocating resources rather than using them more effectively.

Thank you for the clarification and the additional experiments. Together, they make the paper’s conclusions about recall capacity more convincing. I will maintain my original rating of 3.

Comments on the paper

1 comment

As the discussion period draws to a close, we would like to summarize the main additions prompted by the reviews. We have also uploaded a revised manuscript correcting minor issues and restoring the paragraph in §4.3 that relates our capacity-scaling results to prior work on Mamba.

The new MQAR experiments in §4.2 show that recall scales primarily with recurrent state size, with comparatively little change when model width and head layout vary at fixed state size. Section 5 connects the synthetic MQAR finding to language modeling through copying experiments that progress from random sequences to natural text and pretrained models. Before copy fine-tuning, a pretrained linear model gains little from seeing a long passage it is asked to reproduce. After fine-tuning, it copies the passage accurately under teacher forcing at the same state size. Its next-token accuracy falls when the source is absent, motivating interventions that strengthen recall while retaining the language-modeling objective.

Sections 6–7 and Appendix H compare repetition rates, sentence and full-sequence repetition, and methods for reweighting the loss on unchanged text. Sentence repetition and Thresholded LongCE (TLCE), which upweights predictions that benefit from longer context, yield complementary gains. Together, they close the long-distance AR-hit loss gap to standard-trained softmax across all evaluated linear mixers. Experiments at a smaller model scale broadly reproduce the intervention trends, and applying both interventions throughout KDA training brings document extraction close to standard-trained softmax at a language-modeling cost comparable to applying them only during cooldown. The expanded needle-in-a-haystack results (§7.1 and Appendix J) clarify the limits of transfer. Repetition helps most on word retrieval with exact-prefix prompts, and complete-answer accuracy can vary sharply with context length even for the same model and prompt.

These additions address two questions raised across the reviews. The pretrained copying experiments show directly that a language-model checkpoint can learn substantially stronger passage retrieval within its existing state and establish a firmer connection between the synthetic tasks and language. The loss-reweighting experiments show that recall gains do not require replacing fresh training text. Softmax attention also gains recall at a language-modeling cost despite its growing KV cache, so this tradeoff can arise without a fixed-size recurrent state. Together, these findings support our central claim that standard pretraining leaves substantial recall capability underdeveloped at the state sizes we study.

We would welcome any remaining questions or points the reviewers would like us to clarify before the discussion closes. Thank you again to the reviewers and area chairs for the time and care you have given this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.