- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes LCRA, which combines fixed-window local FoX attention with two forms of remote recurrent memory. A hierarchical codebook maps keys and queries to soft routing distributions. The coarse branch accumulates route-conditioned value sums and masses, while a grouped fast-weight branch uses Gated DeltaNet-2-style updates with value targets defined relative to the pre-write coarse estimate. The method aims to improve long-context retrieval without a remote state that grows with context length. The main experiments compare approximately 720M-parameter models trained on about 20B tokens, with a separate smaller-model, 3B-token ablation study.
Critical strengths
- The combination of coarse statistics and group-specific residual memory is a plausible way to organize bounded associative memory.
Critical weaknesses
-
The aspect I found most appealing was the organization of recurrent memory into separately addressable groups. However, this idea has clear precedents. In particular, Mixture-of-Memories (https://arxiv.org/html/2502.13685v1) already routes tokens to multiple independent recurrent states and evaluates this design with Gated DeltaNet. Transformer-VQ (https://arxiv.org/html/2502.13685v1) and Gated Slot Attention (https://arxiv.org/html/2502.13685v1) also provide relevant precedents for organizing bounded memory into slots. This narrows the potentially novel contribution to the specific coupling of hierarchical coarse statistics with grouped fast-weight memories trained on coarse-conditioned residual targets. I find this combination plausible, but the current experiments do not establish its advantage over simpler routed-memory designs. A comparison with a MoM-style or grouped GDN-2 baseline, holding the local branch and persistent-state budget fixed, would help determine whether the proposed coupling offers a substantive improvement. Without such controls, the methodological contribution appears more incremental than the slot-based memory organization initially suggests.
-
The proposed advantage over linear attention is not isolated from memory capacity and architecture changes. Expanding Eqs. (2)-(3) gives the coarse branch an exact normalized linear-attention form, with a learned routing feature map:
kappa(q_t, k_n) = sum_{g,j} w^q_{t,g,j} w^k_{n,g,j}.
Thus, the coarse branch is a structured feature-map choice within normalized linear attention, rather than a fundamentally different computational primitive. The complete architecture adds grouped delta-rule states and local softmax attention, so this observation does not establish equivalence of the entire model to additive linear attention. It does make the appropriate baseline important. LCRA stores 20,544 remote-state scalars per head, excluding local and auxiliary buffers. The paper acknowledges that the compared models are not matched in state size, local-window size, and compute. Matching trainable parameter counts by widening the FFN does not match recurrent-memory capacity. I would like to see a larger-state GDN-2 plus the same local branch, as well as a grouped or routed GDN-2 control, with comparisons under equal persistent-state budgets and a quality-versus-cost curve.
- The ablations do not establish the necessity of the defining design choices.
Removing residual memory also removes its state capacity. This shows that the full configuration is useful, but does not isolate the value of its residual parameterization. The full-value-target variant is a better control because it retains the architecture and capacity, yet its results are mixed: it improves 8K S-NIAH-1 from 26.17% to 30.47% and slightly improves FineWeb perplexity, while hurting 4K retrieval and some other tasks. The paper also does not separately isolate hierarchical routing. Flat routing with the same number of coarse slots, uniform or fixed group assignments, and grouped memory without coarse-conditioned targets would help identify what is responsible for the gains. These targeted controls matter more than adding many loosely related baselines.
Likewise, there is no controlled replacement of local FoX with ordinary sliding-window attention. FoX changes attention weights, not KV compression, so it is not redundant merely because distant history is compressed. Nevertheless, its value in a 64-token window is unproven here. Removing only the forgetting bias while retaining the Pro components, followed by an ordinary SWA baseline with an appropriate positional mechanism, would separate these effects. The remote-only comparison also changes write timing and removes fusion, as the authors acknowledge.
Questions for authors
- Does hierarchical coarse/residual coupling outperform a grouped GDN-2 or MoM-style model when persistent-state capacity and the local branch are held fixed?
- What evidence shows that coarse-conditioned residual targets reduce interference beyond using multiple memory groups? Please include collision/occupancy measurements or controlled associative-recall tests.
Response to Reviewer fMRF
Thank you for highlighting MoM, Transformer-VQ, and Gated Slot Attention as precedents for routed and slot-based memory.
Q1. Coarse–residual coupling and the state budget
LCRA couples routed coarse averages with grouped fast-weight memory. Each group uses the difference between a token’s value and its pre-write coarse estimate as the write target, allowing it to learn key-dependent corrections to the coarse average. We agree that Eqs. (2)–(3) express the coarse readout as normalized linear attention over the remote prefix, with the inner product of the read and write routing weights as its kernel. The walkthrough after Figure 1 and the state table in Appendix A.1 clarify how the two memories interact.
LCRA achieves 95.70% and 76.95% on 8K and 16K S-NIAH-1, compared with 67.19% and 19.53% for GDN-2 in the reported configurations (Table 2). Tables 13–14 report throughput, latency, and peak memory for the evaluated configurations. The reported comparisons leave open how LCRA performs against a larger-state GDN-2 or a grouped GDN-2/MoM-style memory with the same Local FoX branch and a matched persistent-state budget.
Q2. Residual write targets and interference
Table 5 compares the two write targets at fixed architecture and state capacity. Residual and full-value targets score 62.89% and 50.39% on 4K S-NIAH-1, and 26.17% and 30.47% at 8K, respectively. The effect of the write target depends on context length.
Across 2K–8K PG-19 prefixes, the median layer–head pair uses about five effective groups out of eight for both reads and writes (Appendix D.1, Table 11 and Figure 5). Removing the residual readout at the final input position of layer 10 raises next-token NLL by 0.142 nats/token at 8K (Figure 3), showing a contribution to prediction in the trained model. A controlled associative-recall test that varies competition within a group at fixed routing and capacity would directly test the interference hypothesis.
Hierarchical routing and Local FoX
We agree that flat routing at the same slot count would isolate the effect of the hierarchy, and that removing only the forgetting bias would isolate its effect within Local FoX. Section 6 interprets the remote-only ablation at the architecture level because it also changes write timing and fusion.