acceptodds
Under review as a conference paper at ICLR 2027

Lattice: Learning to Efficiently Compress the Memory

Abstract

Attention mechanisms have revolutionized sequence learning but remain bottlenecked by quadratic computational complexity. This paper introduces Lattice, a novel recurrent neural network (RNN) mechanism that leverages the inherent low-rank structure of key-value matrices to efficiently compress contextual information into a fixed number of memory slots, achieving sub-quadratic complexity. By framing this compression as an online optimization problem, we derive a dynamic memory update rule based on a single gradient descent step. The resulting recurrence features a state- and input-dependent gating mechanism, offering an interpretable and theoretically grounded memory update process. The core innovation is the orthogonal update: each memory slot is updated exclusively with information orthogonal to its current state, hence incorporating only novel, non-redundant data to minimize interference with previously stored information. To ensure hardware-efficient training, we derive specialized chunk-wise parallelization forms tailored to this non-linear recurrence. Empirically, Lattice outperforms strong baselines on language modeling and associative recall tasks across diverse context lengths and model sizes, achieving superior memory efficiency.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Orthogonal Updates, RNN, Online Optimization, Memory Compression, Efficient LLM, Test-Time Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Z8LYOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Lattice, a recurrent sequence-mixing layer that compresses streaming context into a fixed-size, matrix-valued memory. The authors formulate memory compression as an online optimization problem; a single gradient-descent step yields the update rule Orthogonal State Recurrence (OSR): each memory slot is updated only with the input component orthogonal to its current content, followed by normalization. The authors show this update is equivalent to Riemannian gradient descent with retraction. Because the recurrence is non-linear, they derive chunk-wise parallel forms (zeroth- and first-order approximations) for hardware-efficient training. Experiments cover language modeling (110M, Books/Pile, contexts up to 16K), zero-shot commonsense reasoning (340M/760M), synthetic MQAR recall, ablations, and throughput, with consistent gains over strong sub-quadratic baselines (TTT, DeltaNet, Gated-DeltaNet, Mamba-2, GLA, Transformer++).

Critical strengths
  • Principled, interpretable derivation: the orthogonal update follows from one online gradient-descent step on a normalized-state reconstruction loss, with clear connections to the delta rule and Riemannian optimization. The unified online-optimization view of existing linear RNNs (Table 3) is a valuable side contribution.
  • Clean ablations isolating the orthogonal update, the normalization, and the memory size.
Critical weaknesses

W1: The core recurrence cannot be parallelized over the time dimension, which limits the practicality of the innovation. Because the state update is non-linear in the state, training cannot use associative scans over the sequence; the authors themselves note that this non-linearity prevents inter-chunk parallelization (Appendix D.3, Future Works). The chunk-wise forms (13) and (15) parallelize computation only within each chunk, so training remains a sequential scan over chunks, and the exact C=1C = 1 recurrence used in all long-context experiments is fully sequential. Competitive training throughput requires large chunks (C∈{16,64}C \in \{16, 64\}; Fig. 9), whose quality cost is characterized only at 4K context (W5). Lattice thereby forfeits the sequence-level parallelism that makes linear RNNs such as Mamba-2 and Gated-DeltaNet hardware-efficient.

W2: No evaluation on real retrieval tasks, which measure the paper's core capability of recalling dense information from natural text. The only recall-oriented evidence is the synthetic MQAR task on 2-layer toy models (artificial 8192-symbol vocabulary, at most 2K tokens). Synthetic MQAR removes what makes recall hard on natural text: keys overlap and interfere, associations are redundant and distributed, and the context carries real statistics.

W3: No long-context evaluation. The long-context evidence is limited to perplexity, which reflects local prediction quality and cannot attribute gains to memory compression. Retrieval, aggregation, and multi-hop reasoning are never evaluated at trained context lengths, and a fixed-size compressed memory is most likely to fail exactly on these capabilities. Recent work of this class (GDN2, arXiv:2605.22791) provides ready benchmarks for this: it evaluates these capabilities with RULER matched to trained context lengths, and evaluates natural-text retrieval with the SWDE/FDA/SQuAD suite. I recommend adding RULER, and, to address W2, that retrieval suite.

W4: Model size and context length are never tested jointly: all 8K/16K results come from the 110M model, and apart from the 4K parameter-scaling curve (Fig. 2), the 340M/760M models were evaluated only on commonsense reasoning (Table 1). The core 4× memory-efficiency claim (m=d/4m = d/4 Lattice outperforming m=dm = d TTT) is validated in this single 110M setting.

W5: The chunk-wise approximations are characterized only at 4K context: all C>1C > 1 results come from 4K-trained models, all long-context experiments use the exact C=1C = 1 recurrence, and the approximation error is never analyzed as chunk size or context length grows.

Questions for authors
  1. (Essential) Retrieval tasks: with 2K-token inputs, evaluate and report average accuracy on real retrieval tasks — SWDE, FDA, SQuAD (TriviaQA/DROP/NQ optional) — with Gated DeltaNet, TTT, Mamba-2, and Transformer++ as baselines; at least at 110M, ideally at 340M/760M.

  2. (Essential) Full RULER (all 13 subtasks), matched to trained context lengths: the 8 retrieval variants (single-key NIAH-1/2/3, multi-key NIAH-1/2/3, multi-value, multi-query), VT, CWE, FWE, and QA-1 (SQuAD) / QA-2 (HotpotQA). Evaluate each model at its trained length (4K-trained models at ≥4K) and report the RULER average per context length.

  3. (Nice-to-have) Average perplexity of the 340M/760M models on WikiText/LAMBADA, and a completed commonsense suite (add OpenBookQA, SIQA, LAMBADA-acc).

4 comments

We thank the Reviewer for the thorough, constructive review and for highlighting our principled derivation, the unified online-optimization view, and our ablations. In the revised manuscript, we have included:

  1. Real-world retrieval (SWDE, SQuAD, FDA): Appendix D.3.1, Table 8.
  2. 16K LongBench (12 tasks) + 4K$\to$20K Extrapolation + RULER : Appendix D.2 (Table 7, Figure 7) and Appendix D.3.3.
  3. WikiText-103 / LAMBADA perplexity, LAMBADA accuracy, OpenBookQA and Social IQA: Appendix D.1, Table 5.
  4. 340M C=16 training (Table 1), 760M C=4 evaluation up to 16K–20K (Table 7, Figure 7), and O(1)\mathcal{O}(1) inference throughput across scales & lengths (Appendix D.5).

We respond to each point below:

W2 & Q1: "No evaluation on real retrieval tasks... with 2K-token inputs, evaluate and report average accuracy on real retrieval tasks — SWDE, FDA, SQuAD... with Gated DeltaNet, TTT, Mamba-2, and Transformer++ as baselines"

We evaluated all eight 760M models on natural-text retrieval (Appendix D.3.1, Table 8): SWDE and SQD (SQuAD), and FDA:

760M Model (30B tok)SWDESQDFDARetrieval Avg (Tab. 8) ↑\uparrow
Transformer++ (Uncompressed KV)67.6933.6535.1245.48
Mamba226.7326.177.2620.06
DeltaNet28.8023.539.9820.77
TTT31.1426.7414.5224.13
GLA32.5828.2515.7925.54
Gated-DeltaNet36.3628.1512.7025.74
Lattice-DEC (Ours)35.1928.3215.3426.28
Lattice-DOT (Ours)36.8128.9913.5226.44

At matched state size, Lattice-DOT (Lattice-DEC) ranks #1 (#2) on average among all recurrent models, with Lattice-DOT outperforming every recurrent baseline on both SWDE and SQuAD. We also updated synthetic MQAR with a full forget-gate τ\tau sweep (Figures 3 & 8, Appendix D.3.2).

W3 & Q2: No long-context evaluation and RULER

For long-context studies, we added two complementary long-context evaluations on all 760M models:

  1. Real-world 16K LongBench & 4K$\to$20K Length Extrapolation (6 tasks) (Appendix D.2, Table 7 & Figure 7): Testing single-doc QA, multi-hop QA (HotpotQA, 2WikiMultihopQA, MuSiQue), summarization, and few-shot ICL at 4×4\times the 4K training length (16K), plus cumulative perplexity up to 20K:
  2. RULER (Appendix D.3.3, Tables 9–12): Reporting both the GDN2 4-task NIAH subset (S-NIAH-1/2 at 1K/2K/4K and S-NIAH-3, MK-NIAH-1 at 1K/2K/4K, Table 9) and other RULER subtasks (S-1..3, MK-1, MV, MQ, VT, CWE, FWE, QA-1, QA-2 at 2K/4K, Table 10). Because non-instruction-tuned 760M base models often retain the target value in state but fail strict surface-format generation, we also report a teacher-forced log-likelihood candidate ranking probe (Top-1) scoring the gold target against 10 unseen format-matched distractors (Tables 11–12).
760M ModelRULER 4-NIAH 1K / 2K / 4K (Tab. 9)4K Ranking Top-1 4-NIAH 1K / 2K / 4KRULER 11-tasks 2K / 4K (Tab. 10)16K LongBench 12-Task (Tab. 7) ↑\uparrow
Transformer++70.4 / 67.3 / 42.998.0 / 98.5 / 87.037.9 / 25.65.9
Mamba240.5 / 14.9 / 7.597.0 / 95.3 / 85.813.6 / 9.311.9
GLA48.3 / 17.2 / 9.195.5 / 92.0 / 83.513.2 / 9.414.0
DeltaNet73.6 / 43.5 / 18.393.8 / 95.3 / 90.522.4 / 10.64.7
Gated-DeltaNet41.5 / 26.8 / 16.495.0 / 95.0 / 96.518.7 / 14.715.0
TTT (no forget)39.5 / 35.4 / 32.997.0 / 96.5 / 98.023.5 / 22.713.6
Lattice-DOT64.9 / 46.1 / 18.397.0 / 96.3 / 97.027.3 / 15.715.4
Lattice-DEC63.8 / 40.8 / 33.696.5 / 95.8 / 97.324.0 / 21.316.0

Key takeaways:

(i) On 16K LongBench (Table 7), Lattice-DEC and Lattice-DOT rank #1 and #2 across all models (while 4K-trained Transformer++ degrades to 5.9), and achieve the lowest perplexity across all 6 benchmarks from 6K to 20K.

(ii) On greedy RULER, the 4-NIAH suite in Table 9, Lattice-DOT achieves the highest recurrent 4-task average at 2K, while Lattice-DEC achieves the highest recurrent average at the 4K training horizon (and both achieve the second highest recurrent average at the 1K). At 4K, Lattice-DEC leads all recurrent models on S-NIAH-3 and MK-NIAH-1. Across the 11-task RULER evaluation (Table 10), Lattice-DOT and Lattice-DEC rank first and second among recurrent architectures at 2K context length.

(iii) Log-likelihood candidate ranking (Tables 11–12) , confirms Lattice retains needles at 94%–100% Top-1 accuracy at 4K (100% on VT and FWE).

W1: "The core recurrence cannot be parallelized over the time dimension... training remains a sequential scan over chunks... Lattice thereby forfeits the sequence-level parallelism that makes linear RNNs such as Mamba-2 and Gated-DeltaNet hardware-efficient."

We agree that state-update non-linearity precludes an associative scan across chunks, so Eqs. (13) and (15) parallelize computation within each chunk while scanning sequentially across chunk boundaries, as noted in Appendix D.5 This property is shared across the non-linear test-time-training (TTT) family (Sun et al., 2024). The primary goal of this work is to introduce orthogonal state recurrence and establish its effectiveness; we do not claim that Lattice matches the training speed of linear RNNs equipped with custom hardware kernels (though the chunkwise form in pure JAX at C=64C=64, Figure 10 shows Lattice scales comparably to TTT and Mamba-2 across 2K–16K and surpasses FlashAttention-2 past 8K). Techniques for scaling TTT recurrences to much larger chunks—such as LaCT (Zhang et al., 2025) and hierarchical chunking in TNT (Li et al., 2025)—are complementary and, alongside custom Triton/Pallas kernels, represent an active direction for future work (Appendix D.5).

At the same time, a key advantage of recurrent models is constant O(1)\mathcal{O}(1) per-token compute and state memory at inference, independent of context length. Appendix D.5 (Tables 14–15, Figure 11) shows that Lattice's inference throughput is strictly on par with Mamba-2, DeltaNet, and TTT across all three model scales (110M, 340M, 760M) and sequence lengths up to 16K, and orders of magnitude faster than Attention.

W4: "Model size and context length are never tested jointly... The core 4×4\times memory-efficiency claim (m=16m=16 Lattice outperforming m=64m=64 TTT) is validated in this single 110M setting."

Our 760M models are now evaluated across context length on 16K LongBench (Table 7), 4K20K length extrapolation (Figure 7), real-world retrieval (Table 8), RULER (Tables 9–12), and an expanded language-modeling and reasoning suite (Table 5). Pretraining additional 340M/760M models at 8K/16K from scratch was outside our rebuttal compute budget. On the memory-efficiency claim, our memory-slot study (m∈{16,32,64,128}m \in \{16, 32, 64, 128\}; Table 2) was conducted as an ablation at the smallest mode scale (110M) for compute reasons.

W5: "The chunk-wise approximations are characterized only at 4K context…., all long-context experiments use the exact recurrence,..."

  • Chunked models tested up to 16K and 20K (760M): The 760M Lattice models are trained with C=4. Evaluated up to 20K tokens, (5×5\times the training length, or 5,120 sequential chunks), they achieve the lowest cumulative perplexity among all models at every cutoff from 6K to 20K (Figure 7) —whereas Transformer++ diverges past 4K and un-gated TTT degrades beyond 8K–10K—, as well as the highest overall averages on 16K LongBench (Table 7). Geometrically, per-token unit-sphere retraction βi,t=(1+∥Δsi,t∥2)−1/2<1\beta_{i,t} = (1+\|\Delta s_{i,t}\|^2)^{-1/2} < 1 (Eq. 9, Table 13) strictly bounds the states at every step, preventing intra-chunk linearization residuals from compounding unboundedly over long horizons.
  • Chunk scaling to C=16 at 110M & 340M: Tables 2 & 13 report C = 16 at 110M. Also, we added a new 340M Lattice at C=16 achieving 48.81 average accuracy (within 0.24–0.33 pts of C=1 and C=4), outperforming every 340M baseline in overall average accuracy as well as on ARC-e and CSQA.

Q3: "Average perplexity on WikiText/LAMBADA, and a completed commonsense suite."

In Appendix D.1 (Table 5), we added WikiText-103 and LAMBADA perplexity plus LAMBADA, OpenBookQA, and Social IQA accuracy for all eight 760M models:

760M Model (30B tok)Wiki-103 PPL ↓\downarrowLAMBADA PPL ↓\downarrowLAMBADA Acc ↑\uparrowOBQA ↑\uparrowSIQA ↑\uparrowAvg (Tab. 5) ↑\uparrow
Transformer++18.1215.7443.4737.8040.3348.52
DeltaNet19.3927.4031.7340.0040.2847.14
Mamba218.6218.0040.6239.2040.3348.59
TTT18.2218.3539.9239.0040.7948.66
GLA18.1117.5440.9539.2039.6148.78
Gated-DeltaNet17.9418.1539.8840.2041.2048.98
Lattice-DEC (Ours)17.6915.9242.3440.4041.1549.87
Lattice-DOT (Ours)17.7916.1841.7239.8042.2750.20

Across all newly added evaluations in Table 5, Lattice achieves the best recurrent score on every task—leading all models overall on WikiText-103, OBQA, SIQA and the 10-task average (50.20 Lattice-DOT, 49.87 Lattice-DEC), while outperforming all recurrent baselines on LAMBADA.

We hope these comprehensive evaluations resolve your concerns.

Dear Reviewer,

Thank you once again for the time and care you put into reviewing our work—your concrete benchmark suggestions motivated extensive new experiments across 760M real-world retrieval, 16K LongBench, 20K length extrapolation, RULER, expanded LM/reasoning tasks and 340M (C=16) training. As the discussion phase wraps up soon, we would like to hear whether our rebuttal and manuscript updates address your questions.

Thanks for the authors' response which have addressed a large proportion of my concerns. However I believe that even chunk recurrence without recovering the capability for computing the linear attention output of different sequential chunks in parallel is not friendly under long context scenarios where context parallelism is an urgent need, and existing context parallel algorithms such as LASP, LASP2 and ZeCO could not be applied in this work. Therefore, I'm more inclined to give a weak rejection.

Reviewer_WN1dOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary
  • Lattice derives a recurrent memory update from an online compression objective with normalized state vectors. Its main idea is to update each slot along a direction orthogonal to its current contents.
Critical strengths
  • The update has a clear geometric interpretation, and the component ablations make the contributions of orthogonalization and normalization reasonably easy to follow. The memory-size experiments and recall results provide useful evidence beyond average language-model scores.
Critical weaknesses
  • The practical computation uses chunk-wise approximations to the nonlinear recurrence. Table 2 already shows a loss in quality as the chunk size increases. I would like quality and throughput reported together for the same variants and chunk sizes, so that the cost of this approximation is visible.

  • The pure-JAX recurrent comparisons in Appendix D.3 are informative about that implementation, but do not establish an advantage over optimized recurrent kernels. The claim of fourfold memory compression would also be stronger with memory-size sweeps for TTT and Gated DeltaNet, rather than comparing against their standard state size.

Questions for authors
  • Orthogonality to the current slot does not by itself guarantee preservation of earlier key-value associations. Can the authors measure interference on previously stored associations after new writes, to test the proposed explanation more directly?
3 comments

We thank the reviewer for the positive assessment and for the precise, constructive suggestions. We address each point below.

W1: "The practical computation uses chunk-wise approximations to the nonlinear recurrence. Table 2 already shows a loss in quality as the chunk size increases. I would like quality and throughput reported together for the same variants and chunk sizes, so that the cost of this approximation is visible."

Thank you for this suggestion. While we reported both quality (Tables 1, 2, & 13) and training throughput (Appendix D.5, Figure 10, formerly Appendix D.3) across C∈{4,16}C \in \{4, 16\} for both per-token and per-chunk normalization, below we combine the exact 110M FineWeb-Edu perplexities (Tables 2 & 13), 340M 7-task zero-shot commonsense accuracies (Table 1, including a new 340M C=16C=16 run), and 110M 2×2 TPU Trillium training throughputs (Figure 10) in one table (which we will add to Appendix D.5):

Model / VariantCC110M PPL (Eq. 13 / 15) ↓\downarrow340M Avg (%) ↑\uparrow110M Train k tok/s (2K / 4K / 8K / 16K) ↑\uparrow
Transformer++ (Pallas FA2)——46.85342 / 211 / 119 / 65
TTT1 / 415.60 (CC=1)47.88176 / 132 / 102 / 70 (CC=4); 310 / 233 / 218 / 189 (CC=16)
Lattice (exact, Eq. 10)115.0249.05—
Lattice (per-token)415.15 / 15.1249.14125 / 107 / 84 / 58
Lattice (per-chunk)415.17 / 15.13—144 / 116 / 92 / 61
Lattice (per-token)1615.30 / 15.2748.81184 / 184 / 161 / 123
Lattice (per-chunk)1615.31 / 15.38—283 / 245 / 216 / 172
  • At 340M, Lattice-DEC at C=16 (48.81%) remains within 0.24 pts (0.49% relative) of C=1 (49.05%) and outperforms every 340M baseline.
  • Applying sphere retraction once per chunk (Eq. 13) (Per-chunk normalization at C=16C=16) yields 15.31 PPL while reaching 245 k tok/s at 4K and 172 k tok/s at 16K—matching pure-JAX TTT and approaching Mamba2 while outpacing FlashAttention-2.

W2a. "The pure-JAX recurrent comparisons in Appendix D.3 are informative about that implementation, but do not establish an advantage over optimized recurrent kernels."

We agree, and we do not claim such an advantage. That comparison (Appendix D.5, formerly D.3) isolates algorithmic chunk-wise scaling under an identical compiler stack. The measurements show that: (i) Lattice scales better with sequence length than Transformer++ (with Pallas FA2 kernel), (ii) its training throughput is comparable to TTT and Mamba2 in the same framework (avoiding the C×CC \times C triangular solves of Gated-DeltaNet), and (iii) its O(1)\mathcal{O}(1) autoregressive inference throughput matches the other recurrent models across all three scales and up to 16K context (Tables 14–15, Fig. 11). Our goal here is to introduce Orthogonal State Recurrence (OSR) and establish its effectiveness; techniques for scaling test-time-training recurrences to larger chunks—such as hierarchical chunking in TNT (Li et al., 2025) and LaCT (Zhang et al., 2025)—alongside custom Triton/Pallas kernels—are natural, active directions of research (Appendix D.5).

W2b. "The claim of fourfold memory compression would also be stronger with memory-size sweeps for TTT and Gated DeltaNet, rather than comparing against their standard state size."

Standard TTT and DeltaNet/Gated-DeltaNet use square d×dd \times d per-head states (m=d=64m=d=64), whereas Lattice explicitly formulates an mm-slot dictionary St∈Rd×mS_t \in \mathbb{R}^{d \times m} decoupling mm from dd. Notably, comparing Lattice at m=16m=16 (15.52 PPL) against baselines at m=64m=64 (TTT 15.60, DeltaNet 16.35; Table 2, Fig. 9) is conservative: shrinking baseline key dimensions to dk=m=16d_k = m = 16 reduces state capacity and increases PPL, widening Lattice's equal-budget lead.

Q1: "Orthogonality to the current slot does not by itself guarantee preservation of earlier key-value associations. Can the authors measure interference on previously stored associations after new writes, to test the proposed explanation more directly?"

We agree with this theoretical point. However, enforcing exact orthogonality to all earlier keys or across slots would require expensive Gram–Schmidt orthogonalization and a growing cache (cf. Srivastava et al., 2014, discussed in Appendix B.1). Within a fixed state budget, OSR mitigates interference because: (i) any component of vtv_t already explained by memory is subtracted (et=ϕ(St−1)kt−vt→0e_t = \phi(S_{t-1})k_t - v_t \to 0) before updating any slot (ΔSt→0,βi,t→1\Delta S_t \to 0, \beta_{i,t} \to 1); and (ii) projecting onto si,t−1⊥s_{i,t-1}^\perp prevents recurring features from inflating slot norms or overwriting unrelated directions within that slot.

Whether this mechanism preserves earlier associations in practice is ultimately an empirical question, which we evaluate directly across recall, long-context understanding, and extrapolation:

  • Post-write interference on MQAR (Appendix D.3.2): All NKVN_{\text{KV}} writes occur before queries, followed by hundreds to thousands of subsequent KV/filler writes. Scaling interfering writes NKV=48→96→256N_{\text{KV}} = 48 \to 96 \to 256 at T=2048T=2048 shows Lattice achieves the highest recall in the tightest compression regimes (d∈{32,64}d \in \{32, 64\}); moreover, doubling intervening writes (T=1024→2048T = 1024 \to 2048 at NKV=256,d=64N_{\text{KV}}=256, d=64) degrades GDN (~71% -> ~64%) and DeltaNet (~61% -> ~43%), whereas Lattice holds steady at ~83%.

  • Real-world retrieval, 16K LongBench and & 20K long context (Tables 5, 7–8, Figure 7): At 760M, Lattice-DOT and Lattice-DEC rank #1 and #2 among all recurrent models on real-world retrieval, 16K LongBench, and 10-task reasoning, while achieving the lowest extrapolated PPL through 20K (Fig. 7).

Moreover, in the revised manuscript, we have included:

  1. 340M C=16C=16 training* (Table 1) and *O(1)\mathcal{O}(1) inference throughput across scales & lengths (Appendix D.5, Tables 14–15, Fig. 11).
  2. Updated MQAR recall (Appendix D.3.2).
  3. 760M real-world retrieval (SWDE, SQuAD, FDA) (Appendix D.3.1, Table 8) and expanded 760M LM & 10-task reasoning (Appendix D.1, Table 5).
  4. 760M 16K LongBench (12 tasks), 4K$\to$20K extrapolation (Appendix D.2, Table 7, Fig. 7), and RULER (Appendix D.3.3, Tables 9–12).

We thank the Reviewer again for the constructive feedback and welcome any further questions during the discussion period.

Dear Reviewer,

Thank you again for your constructive review, helpful suggestions and positive assessment. Before the discussion period concludes, we would like to hear whether our rebuttal and manuscript updates address your questions—we remain happy to clarify any remaining details.

Reviewer_ziZwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Lattice approaches recurrent memory by utilizing online dictionary learning combined with column-normalized memory slots. Subsequently, a gradient step modifies each slot solely with the input component that is orthogonal to it, followed by a renormalization process onto the unit sphere. The study presents chunkwise approximations for training and provides results for perplexity, commonsense (up to 760M), and MQAR.

Critical strengths
  1. The derivation is straightforward: the orthogonal projection directly results from the gradient of a decoder that is normalized by its columns. Additionally, the Riemannian perspective clarifies how normalization functions as an intrinsic forget gate.

  2. Perplexity shows a steady improvement compared to the baselines, with the difference increasing alongside context length. Notably, Lattice using m=16 outperforms TTT with m=64, making it a significant observation.

  3. The approximations made in chunks are meticulously analyzed, investigating the impact of both chunk size and normalization.

Critical weaknesses
  1. The update for each slot essentially follows Oja's rule (Riemannian SGD on the sphere), thus the innovation compared to TTT-style online-optimization models is rather minimal. The assertion regarding "less interference" is applicable only on a per-slot basis, as the update does not remain orthogonal to previous keys or other slots.

  2. The evaluation is quite restricted: models reach up to 760M parameters, commonsense improvements are around 1 point when no seeds are utilized, and the only recall assessment is conducted on synthetic MQAR. The failure of GDN/GLA on MQAR with 256 pairs raises doubts about the tuning of the baselines.

  3. Optimal results are achieved with C=1 or C=4, and throughput is calculated in pure JAX without the use of kernels. Therefore, it remains uncertain if this approach is feasible on a larger scale.

Questions for authors

See weaknesses

4 comments

We thank the reviewer for the constructive assessment and for highlighting our clean derivation. We have updated the revised manuscript with expanded 760M language modeling and 10-task reasoning (Appendix D.1), 16K LongBench and 20K length extrapolation (Appendix D.2), real-world retrieval (Appendix D.3.1), updated MQAR results (Appendix D.3.2), 340M C=16C=16 (Table 1), and inference throughput (Appendix D.5). We respond to each point below.

"The update for each slot essentially follows Oja's rule (Riemannian SGD on the sphere), thus the innovation compared to TTT-style online-optimization models is rather minimal."

We appreciate pointing out the connection to Oja's rule. While both share the same geometry via the orthogonal projector P(si)=I−sisi⊤/∥si∥2P(s_i) = I - s_i s_i^\top / \|s_i\|^2, Lattice differs fundamentally from Oja's rule—and from TTT—in several ways:

  • Coupled key-value compression vs. single-stream PCA: Classical Oja's rule, Δs=η (s⊤x) P(s) x\Delta s = \eta\,(s^\top x)\,P(s)\,x, performs unsupervised variance maximization max⁡∥s∥=112(s⊤xt)2\max_{\|s\|=1} \frac{1}{2}(s^\top x_t)^2 on a single vector stream weighted by self-similarity yi,t=si,t−1⊤xty_{i,t} = s_{i,t-1}^\top x_t (the write strength is the slot's own response). Running m>1m > 1 uncoupled Oja units in parallel causes all slots to collapse onto the same leading eigenvector (rank(S)=1\text{rank}(S)=1) unless O(dm2)\mathcal{O}(d m^2) sequential Gram–Schmidt deflation is added. Lattice instead performs supervised online key-value compression on the oblique manifold OB(d,m)=(Sd−1)m\mathcal{OB}(d,m) = (\mathbb{S}^{d-1})^m (Absil et al., 2009; Boumal, 2023, §7.4), mapping a learned code kt∈Rm↦vt∈Rdk_t \in \mathbb{R}^m \mapsto v_t \in \mathbb{R}^d via Lt=12∥ϕ(S)kt−vt∥22\mathcal{L}_t = \frac{1}{2}\Vert \phi(S) k_t - v_t \Vert_2^2.
    In Lattice-DEC, Δsi,t=−γtki,tP(si,t−1)et\Delta s_{i,t} = -\gamma_t k_{i,t} P(s_{i,t-1}) e_t couples all mm slots through the shared reconstruction residual et=ϕ(St−1)kt−vte_t = \phi(S_{t-1})k_t - v_t, preventing slot collapse in O(dm)\mathcal{O}(dm) time. Classical Oja's rule corresponds to the un-retracted special case where m=1m=1 and kt=st−1⊤vtk_t = s_{t-1}^\top v_t (self-projection rather than a learned key).

  • Exact finite-step retraction: In classical Oja's rule, large finite step sizes push ∣st∣2| s_t |_2 off the unit sphere and can diverge. Lattice applies exact closed-form sphere retraction βi,t=(1+∣Δsi,t∣2)−1/2\beta_{i,t} = (1 + | \Delta s_{i,t} |^2)^{-1/2} (Eq. 9, Prop. 3.1), strictly preserving ∣si,t∣=1|s_{i,t}|=1 and inducing a state- and input-dependent forget gate.

  • State normalization vs. TTT activation normalization (Remarks B.2–B.3, App. B.1): TTT normalizes the output activation, yielding a rank-one Euclidean update with no projection against stored slot content and no control of state norm, leaving StS_t unconstrained in Euclidean space. At equal state size, adding per-slot orthogonal projection and retraction reduces 110M perplexity from 15.60 (TTT) →\to 15.38 →\to 15.07 (Lattice) (Table 2), enabling Lattice with m=16m=16 (15.52) to outperform TTT with m=64m=64 (15.60). Moreover, Lattice's state-dependent gradient does not factorize into a standard rank-1 outer product, requiring our custom chunkwise approximations (Eqs. 13–15).

  • Scope of contribution in context: Landmark models in this line of work have each targeted one core aspect of the recurrence:—for instance, DeltaNet (Yang et al., 2024c) introduced a hardware-efficient chunkwise parallel algorithm for the classical Widrow–Hoff delta rule (Schlag et al., 2021), and Gated-DeltaNet (Yang et al., 2024a) combined DeltaNet with a Mamba-2-style scalar forget gate. In the same spirit, formulating memory compression over unit-norm slots—yielding an orthogonal state recurrence, an intrinsic Riemannian forget gate, and a chunkwise parallel form (Eqs. 13–15)—is a substantive algorithmic contribution at the core recurrence level.

"The evaluation is quite restricted: models reach up to 760M parameters, commonsense improvements are around 1 point when no seeds are utilized..."

  • Single-seed pretraining is standard across GLA, Mamba-2, DeltaNet, TTT, and Gated-DeltaNet, where average gains are typically 0.2–0.9 pts. Our evidence is consistent gains across independently trained model scales (110M, 340M, 760M), model variants, and task families. In Table 1 (760M, 30B tokens)the top four recurrent baselines sit within just 0.39 pts of one another, whereas Lattice-DEC (+0.90) and Lattice-DOT (+1.39) establish a clear lead over Gated-DeltaNet.
    In our expanded 760M evaluation (Appendix D.1, Table 5: adding WikiText-103, LAMBADA, OpenBookQA, and Social IQA), Lattice-DOT (Lattice-DEC) outperforms every baseline—including Transformer++—on 6 of 12 (5 of 12) benchmarks, and beats every recurrent baseline on 9 of 12 (7 of 12) benchmarks (7/10 and 5/10 on zero-shot accuracy). The same lead holds on long-context perplexity (Table 6), 16K LongBench (Table 7), all 6 length-extrapolation benchmarks across 6K–20K (Figure 7), and real-world retrieval (Table 8).

  • Finite-sample evaluation noise: Table 1 totals N=14,126N = 14,126 test questions across 7 independent tasks (Nk∈[1,165,3,270]N_k \in [1{,}165, 3{,}270]). While an individual task has binomial standard error SEk=pk(1−pk)/Nk≈0.8–1.4\text{SE}_k = \sqrt{p_k (1-p_k)/N_k} \approx 0.8\text{–}1.4 pts, averaging across 7 independent tasks reduces the standard error of the mean to SE(Avg)=17∑k=17SEk2≈0.41 pts\text{SE}(\text{Avg}) = \frac{1}{7}\sqrt{\sum_{k=1}^7 \text{SE}_k^2} \approx \mathbf{0.41\text{ pts}}, placing Lattice's +0.90 to +1.39 pt lead at 2.2×2.2\times–3.4×3.4\times SE.

"...and the only recall assessment is conducted on synthetic MQAR."

Beyond synthetic MQAR, we evaluated 760M checkpoints on (i) real-world retrieval (Appendix D.3.1, Table 8: SWDE, SQD and FDA), (ii) LongBench tasks at 16K context (4×4\times the 4K training length; Appendix D.2, Table 7), and (iii) 4K →\to 20K length extrapolation across 6 benchmarks (Appendix D.2, Figure 7):

760M Model (30B tok, 4K train)SWDESQDFDARetrieval Avg (Tab. 8) ↑\uparrow16K LongBench (Tab. 7) ↑\uparrow10-Task Avg (Tab. 5) ↑\uparrow
Transformer++ (Attention)67.6933.6535.1245.485.948.52
Mamba226.7326.177.2620.0611.948.59
DeltaNet28.8023.539.9820.774.747.14
TTT31.1426.7414.5224.1313.648.66
GLA32.5828.2515.7925.5414.048.78
Gated-DeltaNet36.3628.1512.7025.7415.048.98
Lattice-DEC (Ours)35.1928.3215.3426.2816.049.87
Lattice-DOT (Ours)36.8128.9913.5226.4415.450.20

Lattice-DOT and Lattice-DEC rank #1 and #2 among all recurrent models on real-world retrieval (26.44 / 26.28), rank #1 and #2 across all models on 16K LongBench (16.0 / 15.4), and achieve the lowest cumulative perplexity on all 6 benchmarks at every extrapolated cutoff from 6K to 20K (Figure 7).

"The failure of GDN/GLA on MQAR with 256 pairs raises doubts about the tuning of the baselines."

Thank you for raising this. Our initial MQAR runs used the fixed default forget-gate normalizer τ=16\tau = 16 in αt=σ(Wαxt)1/τ\alpha_t = \sigma(W_\alpha x_t)^{1/\tau} from the baseline implementations, which can over-decay early pairs at NKV=256N_{\text{KV}} = 256. In the revised manuscript (Figures 3 & 8, Appendix D.3.2), we swept τ∈{1,2,4,8,16,32}\tau \in \{1, 2, 4, 8, 16, 32\} across models; while tuning τ\tau enables GDN and GLA to improve at larger dimensions, Lattice continues to achieve the strongest recall among recurrent models in the tightest state-compression regimes (NKV=256N_{\text{KV}} = 256 with d=64d = 64 at T∈{1024,2048}T \in \{1024, 2048\}, and d=32d = 32 at T=2048T = 2048).

"Optimal results are achieved with C=1 or C=4, and throughput is calculated in pure JAX without the use of kernels. Therefore, it remains uncertain if this approach is feasible on a larger scale."

  • Larger chunks. Beyond C∈{1,4}C \in \{1,4\}, Tables 2 & 11 show 110M Lattice at C=16C=16, and 340M Lattice-DEC at C=16C=16 in Table 1 outperforming every 340M baseline in ARC-e, CSQA and average accuracy.

  • Inference. A key advantage of recurrent models is a constant per-token cost and memory at inference, independent of context length. App. D.5 shows that Lattice's inference throughput is on par with the other recurrent models at all three scales and different context lengths, and orders of magnitude faster than Attention.

  • Pure-JAX fairness All recurrent baselines were benchmarked in pure JAX (jax.lax.scan) to isolate algorithmic efficiency from custom kernel engineering.

  • Scope. The goal of this work is to introduce the orthogonal state recurrence and establish its effectiveness. Techniques for scaling test-time-training recurrences to much larger chunks, such as LaCT (Zhang et al., 2025) and the hierarchical chunking of Li et al. (2025) [TNT], are complementary and, alongside custom Triton/Pallas kernels, are natural directions for future work (Appendix D.5).

We hope our response and the additional evaluations resolve the reviewer’s concerns and provide a clearer picture of the advantages of the proposed method. We respectfully invite you to reconsider your assessment in light of these clarifications.

Dear Reviewer,

We wanted to thank you again for your thoughtful feedback, which helped us sharpen our positioning relative to Oja's rule, clarify per-slot orthogonality, and expand our empirical evaluation (updated MQAR τ\tau-sweeps at NKV=256N_{\text{KV}}=256, 760M retrieval/long-context benchmarks, 340M C=16C=16, and inference throughput). As the discussion period closes soon, please let us know if our responses and revised manuscript resolve your concerns.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We sincerely thank all Reviewers and the Area Chair for their constructive, insightful feedback. Your suggestions substantially strengthened both the theoretical clarity and empirical breadth of our manuscript.

To provide a consolidated overview as the discussion period concludes, the table below summarizes all new and expanded empirical studies added to the revised manuscript, organized by evaluation goal:

Evaluation GoalNew / Expanded Study & Manuscript ReferenceKey Empirical Takeaway
Language Modeling & Commonsense Reasoning760M Expanded LM & 10-Task Zero-Shot Suite (App. D.1, Table 5): Added WikiText-103 & LAMBADA PPL, plus LAMBADA, OpenBookQA, and Social IQA accuracy alongside Table 1.Lattice-DEC achieves the lowest WikiText-103 PPL across all 8 models; Lattice-DOT and Lattice-DEC rank #1 and #2 overall on 10-task reasoning.
Real-World Factual Retrieval760M Natural-Text Retrieval (App. D.3.1, Table 8): Evaluated structured web extraction (SWDE), extractive QA (SQuAD), and full-document PDF key-value retrieval (FDA).Lattice-DOT and Lattice-DEC rank #1 and #2 among all recurrent models.
Long-Context Understanding & In-Context Learning760M LongBench at 16K (4×4\times training length) (App. D.2, Table 7): 12 tasks across Single-Doc QA, Multi-Hop QA (HotpotQA, 2WikiMQA, MuSiQue), Summarization, and Few-Shot ICL (TREC, TriviaQA, SAMSum).Lattice-DEC (16.0) and Lattice-DOT (15.4) achieve the #1 and #2 overall averages among all models at 16K, with leading few-shot ICL accuracy (TREC, TriviaQA, SAMSum).
Length Extrapolation (4K →\to 20K)760M Zero-Shot Extrapolation up to 5×5\times Horizon (App. D.2, Fig. 7): Cumulative PPL from 4K to 20K on 6 long-context benchmarks (GovReport, QMSum, NarrativeQA, Qasper, CodeParrot, PG-19).Lattice-DEC achieves the lowest perplexity on all 6 benchmarks at every cutoff from 6K to 20K, followed by Lattice-DOT, confirming bounded state dynamics where un-gated TTT drifts past 8K–10K and Transformer++ diverges.
Associative Recall & Post-Write Interference(a) Updated MQAR with Forget-Gate Tuning (App. D.3.2, Fig. 8) across NKV∈{48,96,256}N_{\text{KV}} \in \{48, 96, 256\}, T∈{1024,2048}T \in \{1024, 2048\}; (b) 760M 13-Task RULER & Log-Likelihood State-Retention Probes (App. D.3.3, Tables 9–12).On MQAR (NKV=256N_{\text{KV}}=256), Lattice achieves the highest recall in the tightest regimes (d∈{32,64}d \in \{32, 64\}) and holds steady at ~83% when intervening writes double (T=1024→2048T=1024 \to 2048, while GDN ~71%->~64% and DeltaNet ~61%->~43% degrade). On 760M RULER, Lattice-DEC at 4K and Lattice-DOT 2K lead recurrent models on 4-NIAH.
Memory Efficiency & Chunk-Wise Quality–Speed Trade-Off340M C=16C=16 Pretraining (Table 1) & Joint C∈{1,4,16}C \in \{1, 4, 16\} Throughput Analysis (Tables 2, 13; App. D.5, Fig. 10), complementing mm-slot scaling (Fig. 9).Lattice with 4×4\times fewer slots (m=16m=16) beats full-state (m=64m=64) TTT and DeltaNet at 110M. At 340M, Lattice-DEC at C=16C=16 stays within 0.24 pts of C=1C=1 and beats every 340M baseline.
O(1)\mathcal{O}(1) Autoregressive Inference ScalabilityMulti-Scale (110M–760M) & Multi-Length (2K–16K) Decoding Benchmarks (App. D.5, Tables 14–15, Fig. 11)`.Lattice matches recurrent models across all model sizes and sequence lengths .

We thank the Reviewers and Area Chair once again for helping make this a significantly stronger paper, and we remain happy to address any final questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.