acceptodds
Under review as a conference paper at ICLR 2027

LCRA: Local Attention with Hierarchically Routed Coarse-Residual Memory for Long-Context Retrieval

Abstract

Recurrent language models compress distant history into fixed-size states, but such compression can make individually retrievable associations difficult to preserve as new information accumulates. Existing gated or delta-rule memories primarily focus on how a shared recurrent state is updated, rather than explicitly partitioning memory across associations. Routing-based memories provide an organizational mechanism, but coarse aggregation within a route can still merge distinct values. We introduce LCRA, a Local-Coarse-Residual Architecture that combines fixed-window local FoX attention for recent context with hierarchically routed coarse and residual recurrent memory for distant context. Route-conditioned value-and-mass statistics provide coarse predictions, while grouped fast-weight states use residual targets relative to pre-write coarse estimates. The resulting remote readout is fused with the local branch, while the size of the remote recurrent state remains independent of context length. Across seven approximately 720M-parameter checkpoints, LCRA remains competitive on language modeling and general zero-shot tasks while showing its clearest gains in long-context retrieval. It leads all four evaluated 8K RULER-style synthetic retrieval tasks and retains 76.95% accuracy on 16K S-NIAH-1, compared with 19.53% for GDN-2. LCRA also achieves the lowest fixed-target PG-19 loss through 16K and the highest real-world in-context retrieval mean among the evaluated fixed-state models, while trailing the KV-access references on that mean. A separate 3B-token ablation study supports the complete remote-memory configuration and reveals task- and length-dependent trade-offs in the residual write target. Together, these results position LCRA as a promising fixed-state architecture for long-context retrieval.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-Context Retrieval, Recurrent Language Models, Fixed-State Memory, Hierarchical Routing, Fast-Weight Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fMRFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes LCRA, which combines fixed-window local FoX attention with two forms of remote recurrent memory. A hierarchical codebook maps keys and queries to soft routing distributions. The coarse branch accumulates route-conditioned value sums and masses, while a grouped fast-weight branch uses Gated DeltaNet-2-style updates with value targets defined relative to the pre-write coarse estimate. The method aims to improve long-context retrieval without a remote state that grows with context length. The main experiments compare approximately 720M-parameter models trained on about 20B tokens, with a separate smaller-model, 3B-token ablation study.

Critical strengths
  1. The combination of coarse statistics and group-specific residual memory is a plausible way to organize bounded associative memory.
Critical weaknesses
  1. The aspect I found most appealing was the organization of recurrent memory into separately addressable groups. However, this idea has clear precedents. In particular, Mixture-of-Memories (https://arxiv.org/html/2502.13685v1) already routes tokens to multiple independent recurrent states and evaluates this design with Gated DeltaNet. Transformer-VQ (https://arxiv.org/html/2502.13685v1) and Gated Slot Attention (https://arxiv.org/html/2502.13685v1) also provide relevant precedents for organizing bounded memory into slots. This narrows the potentially novel contribution to the specific coupling of hierarchical coarse statistics with grouped fast-weight memories trained on coarse-conditioned residual targets. I find this combination plausible, but the current experiments do not establish its advantage over simpler routed-memory designs. A comparison with a MoM-style or grouped GDN-2 baseline, holding the local branch and persistent-state budget fixed, would help determine whether the proposed coupling offers a substantive improvement. Without such controls, the methodological contribution appears more incremental than the slot-based memory organization initially suggests.

  2. The proposed advantage over linear attention is not isolated from memory capacity and architecture changes. Expanding Eqs. (2)-(3) gives the coarse branch an exact normalized linear-attention form, with a learned routing feature map:

kappa(q_t, k_n) = sum_{g,j} w^q_{t,g,j} w^k_{n,g,j}.

Thus, the coarse branch is a structured feature-map choice within normalized linear attention, rather than a fundamentally different computational primitive. The complete architecture adds grouped delta-rule states and local softmax attention, so this observation does not establish equivalence of the entire model to additive linear attention. It does make the appropriate baseline important. LCRA stores 20,544 remote-state scalars per head, excluding local and auxiliary buffers. The paper acknowledges that the compared models are not matched in state size, local-window size, and compute. Matching trainable parameter counts by widening the FFN does not match recurrent-memory capacity. I would like to see a larger-state GDN-2 plus the same local branch, as well as a grouped or routed GDN-2 control, with comparisons under equal persistent-state budgets and a quality-versus-cost curve.

  1. The ablations do not establish the necessity of the defining design choices.

Removing residual memory also removes its state capacity. This shows that the full configuration is useful, but does not isolate the value of its residual parameterization. The full-value-target variant is a better control because it retains the architecture and capacity, yet its results are mixed: it improves 8K S-NIAH-1 from 26.17% to 30.47% and slightly improves FineWeb perplexity, while hurting 4K retrieval and some other tasks. The paper also does not separately isolate hierarchical routing. Flat routing with the same number of coarse slots, uniform or fixed group assignments, and grouped memory without coarse-conditioned targets would help identify what is responsible for the gains. These targeted controls matter more than adding many loosely related baselines.

Likewise, there is no controlled replacement of local FoX with ordinary sliding-window attention. FoX changes attention weights, not KV compression, so it is not redundant merely because distant history is compressed. Nevertheless, its value in a 64-token window is unproven here. Removing only the forgetting bias while retaining the Pro components, followed by an ordinary SWA baseline with an appropriate positional mechanism, would separate these effects. The remote-only comparison also changes write timing and removes fusion, as the authors acknowledge.

Questions for authors
  1. Does hierarchical coarse/residual coupling outperform a grouped GDN-2 or MoM-style model when persistent-state capacity and the local branch are held fixed?
  2. What evidence shows that coarse-conditioned residual targets reduce interference beyond using multiple memory groups? Please include collision/occupancy measurements or controlled associative-recall tests.
1 comment
AuthorsOpenReview ↗

Response to Reviewer fMRF

Thank you for highlighting MoM, Transformer-VQ, and Gated Slot Attention as precedents for routed and slot-based memory.

Q1. Coarse–residual coupling and the state budget

LCRA couples routed coarse averages with grouped fast-weight memory. Each group uses the difference between a token’s value and its pre-write coarse estimate as the write target, allowing it to learn key-dependent corrections to the coarse average. We agree that Eqs. (2)–(3) express the coarse readout as normalized linear attention over the remote prefix, with the inner product of the read and write routing weights as its kernel. The walkthrough after Figure 1 and the state table in Appendix A.1 clarify how the two memories interact.

LCRA achieves 95.70% and 76.95% on 8K and 16K S-NIAH-1, compared with 67.19% and 19.53% for GDN-2 in the reported configurations (Table 2). Tables 13–14 report throughput, latency, and peak memory for the evaluated configurations. The reported comparisons leave open how LCRA performs against a larger-state GDN-2 or a grouped GDN-2/MoM-style memory with the same Local FoX branch and a matched persistent-state budget.

Q2. Residual write targets and interference

Table 5 compares the two write targets at fixed architecture and state capacity. Residual and full-value targets score 62.89% and 50.39% on 4K S-NIAH-1, and 26.17% and 30.47% at 8K, respectively. The effect of the write target depends on context length.

Across 2K–8K PG-19 prefixes, the median layer–head pair uses about five effective groups out of eight for both reads and writes (Appendix D.1, Table 11 and Figure 5). Removing the residual readout at the final input position of layer 10 raises next-token NLL by 0.142 nats/token at 8K (Figure 3), showing a contribution to prediction in the trained model. A controlled associative-recall test that varies competition within a group at fixed routing and capacity would directly test the interference hypothesis.

Hierarchical routing and Local FoX

We agree that flat routing at the same slot count would isolate the effect of the hierarchy, and that removing only the forgetting bias would isolate its effect within Local FoX. Section 6 interprets the remote-only ablation at the architecture level because it also changes write timing and fusion.

Reviewer_GbavOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

LCRA combines Local FoX attention for recent tokens with routed coarse and residual memories for older tokens. A token enters remote memory after leaving the local window, and the recurrent state stays fixed in size as context grows. The authors train approximately 720M-parameter models on 20B tokens and run a separate 3B-token ablation study. LCRA improves retrieval over the tested fixed-state models while remaining competitive on language modeling and general tasks.

Critical strengths
  1. The retrieval gains are substantial. LCRA reaches 76.95% on 16K S-NIAH-1, compared with 19.53% for GDN-2. The gains also extend to real-world retrieval: its mean is 32.66% versus 28.77% for GDN-2, with improvements on four of six tasks (Tables 2–3).

  2. The evaluation covers several useful settings. Alongside retrieval, the paper tests language modeling and general tasks. I particularly like that PG-19 uses the same target tokens at each context length. The separate ablation study also compares residual and full-value write targets directly (Tables 4–5; Appendix B).

Critical weaknesses

My main concern is that the ablations leave the benefit of the particular memory design unclear. Removing residual memory reduces 4K S-NIAH-1 from 62.89% to 21.88%, but also removes about 80% of the remote state. Widening the FFN matches parameters without restoring that memory. The full-value-target comparison is more informative and supports the residual objective. What I am still missing is a comparison with a simpler memory using the same Local FoX branch and a similar state budget. This would help show how much the routing and coarse–residual design contribute to the gains (Tables 5–6).

The training cost makes this comparison particularly relevant. At 4K, LCRA runs at 16.240K tokens/s versus 28.482K for GDN-2. LCRA also has lower request latency, though the models use different execution paths. A cost breakdown would help explain this trade-off (Appendix E; Table 12).

Questions for authors
  1. How does LCRA compare with Local FoX combined with a standard delta-rule memory at a similar state budget?

  2. As context grows, how are the memory groups used, and how well does the residual branch correct the coarse predictions?

  3. Which components account for the training cost, and how much do backend differences affect the latency comparison?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Gbav

Q1. Comparison at a matched remote-state budget

Thank you for highlighting the role of the state budget. We agree that a direct comparison with standard delta-rule memory should use the same Local FoX branch and a comparable remote-state budget. In the original ablation, removing the residual branch reduced the remote state from 20,544 to 4,160 scalars per head. Widening the FFN kept parameter counts close (Appendix A, Table 6). The drop from 62.89% to 21.88% on 4K S-NIAH-1 reflects these joint changes (Table 5).

The write-target ablation holds the architecture and state budget fixed. On S-NIAH-1, residual writes score 62.89% at 4K and 26.17% at 8K, compared with 50.39% and 30.47% for full-value writes (Table 5). The relative performance of the two write targets thus varies with context length within this grouped architecture. The proposed control addresses how the full remote memory design compares with standard delta-rule memory at a matched state budget.

Q2. Memory-group usage and residual correction

Group usage. We analyzed routing on 32 PG-19 books using the 255M LCRA checkpoint trained on 3B tokens. From 2K to 8K, the median effective group count across layer–head pairs stayed at about 5.1–5.5 out of eight. Across pairs, the median of the mean per-token top group weight was about 63% for writes and 67% for reads (Appendix D.1, Table 11 and Fig. 5). A typical head distributes its reads and writes across several groups over a prefix. At each token, the top group receives most of the weight on average. For example, at 8K, layer 8, head 2 has an effective read group count of 7.59 and a mean per-token top read weight of 87.1%. In the ten paired retrieval examples in Fig. 6, layer 8, head 5 distributes target writes across groups and usually favors group 6 when reading during answer generation.

Residual correction. In the same trained checkpoint, we used only the coarse remote readout at the final input position of one layer at a time, preserving earlier memory reads and writes. The residual contribution to next-token prediction varied by layer. Layers 8–10 showed the largest NLL increases at all three lengths, and their 95% pointwise paired bootstrap intervals over books were above zero. Their increases at 8K were 0.087, 0.079, and 0.142 nats/token, respectively (Fig. 3). In the 8K self-key probe, all 120 layer–head pairs had positive mean cosine alignment with the coarse error, and 80 had a positive mean relative reduction in squared reconstruction error (Appendix D.2, Figs. 7–8). Alignment measures the direction of the correction. Error reduction also depends on its magnitude.

Q3. Training cost and inference latency

Thank you for suggesting a training cost breakdown and highlighting the effect of execution paths on latency.

Training cost. In the 255M, 4K training profile on an RTX 4090, LCRA's largest additional cost over GDN-2 is in the grouped delta-rule kernels (37.81 ms per step). This increase, together with 19.90 ms in the coarse kernels and 9.26 ms in the local attention kernels, accounts for 78.1% of the gap in cumulative GPU kernel time per step (Appendix F, Figure 9). A preliminary analysis of a single-step trace shows that most of the extra delta-rule time was spent in the backward pass, with hotspots in gradient computation and recomputation of intermediates. We will test saving selected forward intermediates to reduce recomputation and fusing adjacent kernels to reduce tensor reads and writes, measuring training-step time and peak GPU memory.

Inference latency. In Figure 4, LCRA uses CUDA Graph and GDN-2 uses eager cached decoding. In their 4K comparison, GDN-2 has lower full-request latency when both models use eager execution or both use CUDA Graph (Appendix F, Table 15). With CUDA Graph enabled for both, decoding and sampling account for about 75% of the mean latency gap (Appendix F.2), so we will profile these stages first.

Reviewer_WcnaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes LCRA, a linear-complexity architecture for long-context language modeling. LCRA has three main parts: local window attention, coarse memory, and residual memory. The local branch handles recent tokens. The remote branch routes older tokens into coarse memory and uses residual memory to correct the coarse predictions. The paper evaluates LCRA on language modeling, synthetic retrieval, PG-19, and real-world retrieval tasks.

Critical strengths
  1. Strong retrieval results. The S-NIAH-1 results are striking: 95.70% at 8K and 76.95% at 16K, compared with 67.19% and 19.53% for GDN-2. LCRA also has the best real-world retrieval mean among the fixed-state models and the best PG-19 fixed-target loss through 16K.

  2. An interesting multi-memory design. Using separate local, coarse, and residual memory branches is a reasonable way to capture different aspects of the context.

  3. Honest limitations. The paper clearly states several limitations, including single-seed training, non-resource-matched storage groups, the architecture-level nature of the remote-only comparison, and the lack of an isolated hierarchical-routing ablation.

Critical weaknesses
  1. The method is hard to understand. Many concepts are introduced without enough explanation. For example, “coarse memory” and “residual memory” appear in the abstract, but the reader does not get a simple intuition for what they store or why both are needed. The method has many components and symbols, so a step-by-step example would help. I suggest adding:

    • a short walkthrough of one token through the model;
    • a table with the shape and role of each state (A, Z, M, local cache);
    • a clear explanation of why residual memory is needed after coarse memory.
  2. The inference latency comparison may be unfair. Figure 3 compares latency, but CUDA Graph is an important optimization that can greatly reduce latency. It is not clear whether all models use CUDA Graph or only LCRA. If only LCRA uses it, the comparison is not fair. This is especially important because LCRA has lower training throughput than the baselines, so its better inference latency is surprising and needs a clear explanation.

  3. State size and compute are not matched. LCRA uses several kinds of state: a local window cache, coarse memory (A, Z), and residual memory (M). Its larger state may partly explain its better performance [1]. Tables 1 and 2 do not report state size or per-token compute for each model, so the comparisons may be unfair. Please add a table with state size, FLOPs, and memory for all models, or match these resources.

[1] Scaling up the State Size of RNN LLMs for Long-Context Scenarios

Questions for authors

see weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer Wcna

Thank you for the detailed feedback. We revised the method presentation and clarified the scope of the efficiency and resource comparisons.

1. Method clarity

We revised the abstract to explain the roles of the two remote memories. Coarse memory pools values routed to each prototype into a weighted average. When distinct values contribute to the same average, residual memory can learn key-dependent corrections using the difference between a token’s value and its group’s pre-write coarse estimate as the write target.

Section 3 now includes a walkthrough after Figure 1. When computing the output at position 65, LCRA writes token 1, which has just left the 64-token local window. The query at position 65 then reads both updated remote states, and LCRA fuses their readouts with the Local FoX output. The example continues at position 66 to show how this handoff repeats as the sequence grows. Appendix A.1 now gives the per-head shapes and roles of AA, ZZ, MM, and the Local FoX cache, along with the other inference buffers and method notation.

2. Latency and execution paths

Only LCRA used CUDA Graph in the original latency plot, now Figure 4. The revised caption specifies the execution paths. We also compared the 720M LCRA and GDN-2 checkpoints on the same RTX PRO 6000 with 4K input, batch size one, and 128 generated tokens (Appendix F, Table 15). GDN-2 had lower full-request latency in both eager mode (2.741 s vs. 7.118 s) and CUDA Graph mode (0.425 s vs. 1.289 s), with per-request graph preparation included in the Graph timings. The original latency ordering thus depends on the execution paths used. In the separate 255M training profile, grouped delta-rule updates, coarse memory, and Local FoX account for 78.1% of the GPU kernel-time gap (Appendix F, Figure 9).

3. State size and compute

We agree that state capacity and compute can affect retrieval. The main from-scratch comparisons use approximately 720M parameters and 20B training tokens, with architecture-specific state and compute budgets (Section 6). Appendix A reports 20,544 remote recurrent scalars per LCRA head, or 246,528 per layer for the 720M model. The new table in Appendix A.1 also gives the shapes of the local and pending-write buffers. Appendix F reports measured peak allocated GPU memory (Table 14), training throughput, and request latency (Table 13).

Within LCRA, the full-value-target ablation changes the write target at fixed architecture and state capacity (Table 5). The cross-model results describe the performance of the evaluated configurations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

LCRA — Summary of Contributions and Revisions

Dear Area Chair,

Thank you for considering our paper and the reviewer discussion.

Contribution. LCRA combines a 64-token Local FoX window with remote memory whose size stays fixed as context grows. The coarse memory pools routed values, and each residual group uses the difference between a token’s value and its pre-write coarse estimate as its write target. This lets the model learn key-dependent corrections to the pooled values.

Main results. At approximately 720M parameters and 20B training tokens, LCRA leads all four evaluated synthetic retrieval tasks at 8K. On S-NIAH-1, it reaches 95.70% at 8K and 76.95% at 16K, compared with 67.19% and 19.53% for GDN-2 (Table 2). The gains extend to real-world retrieval. LCRA achieves the highest six-task mean among the evaluated fixed-state models (32.66%, Table 3) and the lowest PG-19 loss on matched target tokens at every evaluated length through 16K (Figure 2).

What the revision adds. The walkthrough after Figure 1 shows how a token moves from the local window into remote memory, and Appendix A.1 gives the shapes and roles of the state tensors. The new diagnostics show that reads and writes each cover about five effective groups out of eight for the median layer–head pair across 2K–8K PG-19 inputs (Appendix D.1, Table 11 and Figure 5). They also quantify where the residual readout helps prediction. At 8K, removing it at the final input position of layer 10 increases mean next-token NLL by 0.142 nats/token in the same trained model (Figure 3).

Efficiency. LCRA trains more slowly than GDN-2 and has higher request latency in the matched 4K inference test. The revision documents the execution paths and component costs (Figure 4, Appendix F, Figure 9 and Table 15).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.