- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes Lattice, a recurrent sequence-mixing layer that compresses streaming context into a fixed-size, matrix-valued memory. The authors formulate memory compression as an online optimization problem; a single gradient-descent step yields the update rule Orthogonal State Recurrence (OSR): each memory slot is updated only with the input component orthogonal to its current content, followed by normalization. The authors show this update is equivalent to Riemannian gradient descent with retraction. Because the recurrence is non-linear, they derive chunk-wise parallel forms (zeroth- and first-order approximations) for hardware-efficient training. Experiments cover language modeling (110M, Books/Pile, contexts up to 16K), zero-shot commonsense reasoning (340M/760M), synthetic MQAR recall, ablations, and throughput, with consistent gains over strong sub-quadratic baselines (TTT, DeltaNet, Gated-DeltaNet, Mamba-2, GLA, Transformer++).
Critical strengths
- Principled, interpretable derivation: the orthogonal update follows from one online gradient-descent step on a normalized-state reconstruction loss, with clear connections to the delta rule and Riemannian optimization. The unified online-optimization view of existing linear RNNs (Table 3) is a valuable side contribution.
- Clean ablations isolating the orthogonal update, the normalization, and the memory size.
Critical weaknesses
W1: The core recurrence cannot be parallelized over the time dimension, which limits the practicality of the innovation. Because the state update is non-linear in the state, training cannot use associative scans over the sequence; the authors themselves note that this non-linearity prevents inter-chunk parallelization (Appendix D.3, Future Works). The chunk-wise forms (13) and (15) parallelize computation only within each chunk, so training remains a sequential scan over chunks, and the exact recurrence used in all long-context experiments is fully sequential. Competitive training throughput requires large chunks (; Fig. 9), whose quality cost is characterized only at 4K context (W5). Lattice thereby forfeits the sequence-level parallelism that makes linear RNNs such as Mamba-2 and Gated-DeltaNet hardware-efficient.
W2: No evaluation on real retrieval tasks, which measure the paper's core capability of recalling dense information from natural text. The only recall-oriented evidence is the synthetic MQAR task on 2-layer toy models (artificial 8192-symbol vocabulary, at most 2K tokens). Synthetic MQAR removes what makes recall hard on natural text: keys overlap and interfere, associations are redundant and distributed, and the context carries real statistics.
W3: No long-context evaluation. The long-context evidence is limited to perplexity, which reflects local prediction quality and cannot attribute gains to memory compression. Retrieval, aggregation, and multi-hop reasoning are never evaluated at trained context lengths, and a fixed-size compressed memory is most likely to fail exactly on these capabilities. Recent work of this class (GDN2, arXiv:2605.22791) provides ready benchmarks for this: it evaluates these capabilities with RULER matched to trained context lengths, and evaluates natural-text retrieval with the SWDE/FDA/SQuAD suite. I recommend adding RULER, and, to address W2, that retrieval suite.
W4: Model size and context length are never tested jointly: all 8K/16K results come from the 110M model, and apart from the 4K parameter-scaling curve (Fig. 2), the 340M/760M models were evaluated only on commonsense reasoning (Table 1). The core 4× memory-efficiency claim ( Lattice outperforming TTT) is validated in this single 110M setting.
W5: The chunk-wise approximations are characterized only at 4K context: all results come from 4K-trained models, all long-context experiments use the exact recurrence, and the approximation error is never analyzed as chunk size or context length grows.
Questions for authors
-
(Essential) Retrieval tasks: with 2K-token inputs, evaluate and report average accuracy on real retrieval tasks — SWDE, FDA, SQuAD (TriviaQA/DROP/NQ optional) — with Gated DeltaNet, TTT, Mamba-2, and Transformer++ as baselines; at least at 110M, ideally at 340M/760M.
-
(Essential) Full RULER (all 13 subtasks), matched to trained context lengths: the 8 retrieval variants (single-key NIAH-1/2/3, multi-key NIAH-1/2/3, multi-value, multi-query), VT, CWE, FWE, and QA-1 (SQuAD) / QA-2 (HotpotQA). Evaluate each model at its trained length (4K-trained models at ≥4K) and report the RULER average per context length.
-
(Nice-to-have) Average perplexity of the 340M/760M models on WikiText/LAMBADA, and a completed commonsense suite (add OpenBookQA, SIQA, LAMBADA-acc).
4 comments
W1: "The core recurrence cannot be parallelized over the time dimension... training remains a sequential scan over chunks... Lattice thereby forfeits the sequence-level parallelism that makes linear RNNs such as Mamba-2 and Gated-DeltaNet hardware-efficient."
We agree that state-update non-linearity precludes an associative scan across chunks, so Eqs. (13) and (15) parallelize computation within each chunk while scanning sequentially across chunk boundaries, as noted in Appendix D.5
This property is shared across the non-linear test-time-training (TTT) family (Sun et al., 2024). The primary goal of this work is to introduce orthogonal state recurrence and establish its effectiveness; we do not claim that Lattice matches the training speed of linear RNNs equipped with custom hardware kernels (though the chunkwise form in pure JAX at , Figure 10 shows Lattice scales comparably to TTT and Mamba-2 across 2K–16K and surpasses FlashAttention-2 past 8K).
Techniques for scaling TTT recurrences to much larger chunks—such as LaCT (Zhang et al., 2025) and hierarchical chunking in TNT (Li et al., 2025)—are complementary and, alongside custom Triton/Pallas kernels, represent an active direction for future work (Appendix D.5).
At the same time, a key advantage of recurrent models is constant per-token compute and state memory at inference, independent of context length. Appendix D.5 (Tables 14–15, Figure 11) shows that Lattice's inference throughput is strictly on par with Mamba-2, DeltaNet, and TTT across all three model scales (110M, 340M, 760M) and sequence lengths up to 16K, and orders of magnitude faster than Attention.
W4: "Model size and context length are never tested jointly... The core memory-efficiency claim ( Lattice outperforming TTT) is validated in this single 110M setting."
Our 760M models are now evaluated across context length on 16K LongBench (Table 7), 4K20K length extrapolation (Figure 7), real-world retrieval (Table 8), RULER (Tables 9–12), and an expanded language-modeling and reasoning suite (Table 5). Pretraining additional 340M/760M models at 8K/16K from scratch was outside our rebuttal compute budget.
On the memory-efficiency claim, our memory-slot study (; Table 2) was conducted as an ablation at the smallest mode scale (110M) for compute reasons.
W5: "The chunk-wise approximations are characterized only at 4K context…., all long-context experiments use the exact recurrence,..."
- Chunked models tested up to 16K and 20K (760M): The 760M Lattice models are trained with C=4. Evaluated up to 20K tokens, ( the training length, or
5,120sequential chunks), they achieve the lowest cumulative perplexity among all models at every cutoff from 6K to 20K (Figure 7) —whereas Transformer++ diverges past 4K and un-gated TTT degrades beyond 8K–10K—, as well as the highest overall averages on 16K LongBench (Table 7). Geometrically, per-token unit-sphere retraction (Eq. 9, Table 13) strictly bounds the states at every step, preventing intra-chunk linearization residuals from compounding unboundedly over long horizons. - Chunk scaling to
C=16at 110M & 340M: Tables 2 & 13 reportC = 16at 110M. Also, we added a new 340MLatticeatC=16achieving48.81average accuracy (within0.24–0.33pts ofC=1andC=4), outperforming every 340M baseline in overall average accuracy as well as onARC-eandCSQA.
Q3: "Average perplexity on WikiText/LAMBADA, and a completed commonsense suite."
In Appendix D.1 (Table 5), we added WikiText-103 and LAMBADA perplexity plus LAMBADA, OpenBookQA, and Social IQA accuracy for all eight 760M models:
| 760M Model (30B tok) | Wiki-103 PPL | LAMBADA PPL | LAMBADA Acc | OBQA | SIQA | Avg (Tab. 5) |
|---|---|---|---|---|---|---|
| Transformer++ | 18.12 | 15.74 | 43.47 | 37.80 | 40.33 | 48.52 |
| DeltaNet | 19.39 | 27.40 | 31.73 | 40.00 | 40.28 | 47.14 |
| Mamba2 | 18.62 | 18.00 | 40.62 | 39.20 | 40.33 | 48.59 |
| TTT | 18.22 | 18.35 | 39.92 | 39.00 | 40.79 | 48.66 |
| GLA | 18.11 | 17.54 | 40.95 | 39.20 | 39.61 | 48.78 |
| Gated-DeltaNet | 17.94 | 18.15 | 39.88 | 40.20 | 41.20 | 48.98 |
| Lattice-DEC (Ours) | 17.69 | 15.92 | 42.34 | 40.40 | 41.15 | 49.87 |
| Lattice-DOT (Ours) | 17.79 | 16.18 | 41.72 | 39.80 | 42.27 | 50.20 |
Across all newly added evaluations in Table 5, Lattice achieves the best recurrent score on every task—leading all models overall on WikiText-103, OBQA, SIQA and the 10-task average (50.20 Lattice-DOT, 49.87 Lattice-DEC), while outperforming all recurrent baselines on LAMBADA.
We hope these comprehensive evaluations resolve your concerns.
Dear Reviewer,
Thank you once again for the time and care you put into reviewing our work—your concrete benchmark suggestions motivated extensive new experiments across 760M real-world retrieval, 16K LongBench, 20K length extrapolation, RULER, expanded LM/reasoning tasks and 340M (C=16) training. As the discussion phase wraps up soon, we would like to hear whether our rebuttal and manuscript updates address your questions.
Thanks for the authors' response which have addressed a large proportion of my concerns. However I believe that even chunk recurrence without recovering the capability for computing the linear attention output of different sequential chunks in parallel is not friendly under long context scenarios where context parallelism is an urgent need, and existing context parallel algorithms such as LASP, LASP2 and ZeCO could not be applied in this work. Therefore, I'm more inclined to give a weak rejection.
We thank the Reviewer for the thorough, constructive review and for highlighting our principled derivation, the unified online-optimization view, and our ablations. In the revised manuscript, we have included:
SWDE,SQuAD,FDA):Appendix D.3.1, Table 8.Appendix D.2 (Table 7, Figure 7)andAppendix D.3.3.Appendix D.1, Table 5.C=16training (Table 1), 760MC=4evaluation up to 16K–20K (Table 7, Figure 7), and O(1) inference throughput across scales & lengths (Appendix D.5).We respond to each point below:
We evaluated all eight 760M models on natural-text retrieval (
Appendix D.3.1, Table 8):SWDEandSQD(SQuAD), andFDA:At matched state size, Lattice-DOT (Lattice-DEC) ranks #1 (#2) on average among all recurrent models, with Lattice-DOT outperforming every recurrent baseline on both
SWDEandSQuAD. We also updated synthetic MQAR with a full forget-gate τ sweep (Figures 3 & 8, Appendix D.3.2).For long-context studies, we added two complementary long-context evaluations on all 760M models:
Appendix D.2, Table 7 & Figure 7): Testing single-doc QA, multi-hop QA (HotpotQA,2WikiMultihopQA,MuSiQue), summarization, and few-shot ICL at 4× the 4K training length (16K), plus cumulative perplexity up to 20K:Appendix D.3.3, Tables 9–12): Reporting both the GDN2 4-task NIAH subset (S-NIAH-1/2 at 1K/2K/4K and S-NIAH-3,MK-NIAH-1at 1K/2K/4K,Table 9) and other RULER subtasks (S-1..3,MK-1,MV,MQ,VT,CWE,FWE,QA-1,QA-2at 2K/4K,Table 10). Because non-instruction-tuned 760M base models often retain the target value in state but fail strict surface-format generation, we also report a teacher-forced log-likelihood candidate ranking probe (Top-1) scoring the gold target against 10 unseen format-matched distractors (Tables 11–12).Key takeaways:
(i) On 16K LongBench (
Table 7), Lattice-DEC and Lattice-DOT rank #1 and #2 across all models (while 4K-trained Transformer++ degrades to5.9), and achieve the lowest perplexity across all 6 benchmarks from 6K to 20K.(ii) On greedy RULER, the
4-NIAHsuite inTable 9, Lattice-DOT achieves the highest recurrent 4-task average at 2K, while Lattice-DEC achieves the highest recurrent average at the 4K training horizon (and both achieve the second highest recurrent average at the 1K). At 4K, Lattice-DEC leads all recurrent models on S-NIAH-3 and MK-NIAH-1. Across the 11-task RULER evaluation (Table 10), Lattice-DOT and Lattice-DEC rank first and second among recurrent architectures at 2K context length.(iii) Log-likelihood candidate ranking (
Tables 11–12) , confirms Lattice retains needles at94%–100%Top-1 accuracy at 4K (100%onVTandFWE).