acceptodds
Under review as a conference paper at ICLR 2027

Latent Lattice Language Models

Abstract

Standard autoregressive transformers route all per-token computation through a single active latent. This latent is responsible for querying and processing the "memory" from the past steps, forming new memory (updating the KV cache) for the future steps, producing residual updates to its own state, and finally decoding into the next token. The memory is a write-once, read-only log of the latent's past states, while all active computation still rests on that one latent vector. Recent work increases the computation performed by this single latent either along depth - by adding more layers or looping - or along time, by letting it persist across steps rather than being decoded at each one. However, the model remains bottlenecked, relying on a single latent to perform every role. We instead ask: how can we increase the computational latent capacity available to a language model, and what does that buy us? To that end, we introduce an additional horizontal latent at each Lattice layer, updated continuously along the temporal dimension. Together with the standard vertical latent that runs along the network's depth axis, these horizontal recurrent states form our Lattice of Latents. Unlike hybrid Transformer-SSM architectures, where recurrent state typically remains internal to a sequence-mixing operator (primarily focused on replacing softmax-attention and KV cache), our horizontal latent acts as an explicit companion computational state to the vertical latent, jointly driving attention, KV-cache writes, MLP computation, and decoding. We propose several mechanisms to increase the expressivity of these horizontal latents while retaining token-parallel teacher-forced computation within each pass. Across pre-training budgets of 10B-30B tokens, Lattice models consistently improve language-modeling quality and downstream transfer. After pre-training on 20B FineWeb-Edu tokens, our 406M-parameter Latent Lattice Language Model lowers perplexity from to relative to a parameter and compute-matched Transformer, while also matching a 609M-parameter, 40-layer Transformer that uses as many parameters, the KV-cache footprint, and as many inference FLOPs. The gains also extend to downstream evaluation and mathematical reasoning after finetuning. Project Page: https://latent-lattice-language-model.netlify.app/

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Recurrent Transformers, Latent Modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ddsBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

I provide the review regarding “Latent Lattice Language Models”. This study provides a novel method to incorporate an RNN-like state and Transformer-like KV cache. The proposed method passes through the latent space along with time and incorporates a KV cache in each layer. The proposed method achieves better performance with fewer parameters and FLOPs than Transformers. The authors also provide detailed ablation test that the proposed components contribute to the performance gain.

Critical strengths

This is a very impressive study, especially in the experimental results. The authors provide a novel method to incorporate RNN and Transformer mechanisms and achieve better performance than Transformers. This is very impressive because generally RNNs and Transformer variants are faster but have lower performance. Transformers are the best method, but they are computationally expensive. The experimental results suggest that the authors' method has better performance with lower computational cost. This is very important because in the past 10 years, Transformers have been the upper bound of neural network architectures, and others have only advantages in task-specific or computational aspects. Figure 4 is very impressive and shows that the proposed components are useful. I feel something great breakthrough in this study.

Critical weaknesses

I find several Weaknesses in this study.

  1. The authors provide apps and a detailed explanation in the main text and appendix. I believe the authors' effort is extensive, and I do not think this is a major fault, but I still cannot understand the details and the whole picture of the proposed method. I can understand each component; however, understanding the connection between each component and the procedure of training and inference is still difficult to follow. Additional figures, videos, and mathematical formulas may help readers.

  2. The baseline PPL of the experiments is not small enough. They are around 10, which is marginally acceptable at an academic level; however, if the authors have the computational resources, I hope they can run experiments on larger LLMs with larger datasets.

  3. In addition, the source code has not been released.

Questions for authors

The proposed method seems to inject RNN latent space into the KV cache, but I could not understand the whole mechanism despite the authors' effort not only main-text but also apps. It seems that the nature of the proposed method is a complicated mechanism, and existing way to express it cannot capture it. I hope the authors provide more figures and mathematical formulas, especially to connect all components for better understanding. I can understand each component and some intuition, but it is still difficult to understand, and I cannot give clear acceptance. However, I provide weak acceptance score because the experimental results are impressive.

2 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer ddsB

We thank the reviewer for their encouraging assessment of our contribution and experimental results. We agree that the original presentation could better connect the components and clarify training and inference. We have reorganized Section 3 and improved the accompanying explanations and figures (Figs. 1–3).

We would also like to highlight the new results in Appendix B (shown in red in the revised manuscript), demonstrating the Lattice's effectiveness across recurrent backbones and its complementarity with existing residual-stream enhancements.

W1 / Q1: Understanding the components and training/inference procedure


Core mechanism (Sec. 3.2, Fig. 1). The Lattice adds a horizontal recurrent latent rtℓr_t^\ell at each Lattice layer. It persists across tokens and assists the vertical residual stream htℓh_t^\ell flowing through depth. Each layer performs two operations:

  1. Update (Sec. 3.2.2, Fig. 3): The horizontal state evolves as rtℓ=gtℓ⊙rt−1ℓ+btℓr_t^\ell = g_t^\ell \odot r_{t-1}^\ell + b_t^\ell, with coefficients computed from the current vertical input htℓ−1h_t^{\ell-1}.
  2. Condition (Sec. 3.2.1, Fig. 2): The updated state modulates the Q, K, V and MLP computations through feature-wise linear modulation (FiLM). It therefore influences attention reads and KV-cache writes, but is not itself appended to the KV cache. All modulation projections are zero-initialized, so at initialization the model computes exactly what the backbone does.

We study three enhancements to the horizontal pathway (Sec. 3.2.2), each ablated in Fig. 4 (Sec. 4.3):

  • Input injection feeds the input token embedding to the horizontal latent at every layer.
  • vstream carries lower-layer horizontal latents up to higher layers.
  • The Jacobi refiner feeds the previous token's top-layer horizontal latent back down to the bottom layer.

Training. The recurrence coefficients depend only on current-token inputs, not on the previous horizontal state. An associative parallel scan therefore computes all horizontal states at each layer across tokens, preserving token-parallel teacher forcing. Input injection and vstream retain this property. The optional non-linear top-to-bottom feedback instead uses three token-parallel Jacobi sweeps (J3), at additional compute cost.

Inference. Generation is autoregressive. At each Lattice layer, we update the horizontal state, use it to condition attention and the MLP, and advance the vertical stream. Each layer maintains one fixed-size recurrent state per sequence and adds no KV-cache entries. In sequential rollout, the optional top-to-bottom feedback is evaluated exactly, without Jacobi sweeps. We report rollout and J3 evaluation separately.

W2: Larger models and datasets


We agree that validation at larger scales is important. Our compute budget currently limits experiments to sub-billion-parameter models. Within that budget, we provide evidence across 10B, 20B and 30B training tokens, including extended post-Chinchilla pretraining:

  • 10B (Fig. 4, Sec. 4.3): A 406M Lattice (15.63 PPL) outperforms 600M Transformers scaled by width (15.98) or by depth (15.82). A 291M Lattice approximately matches a 406M Transformer.
  • 20B (Table 1): At matched ~400M parameters, the Lattice reduces FineWeb-Edu PPL from 15.41 to 14.40.
  • 30B (Table 6, App. A.4): At 400M, the Lattice reduces in-domain and out-of-domain PPL by 5–11%. At 600M, it improves FineWeb-Edu PPL from 13.80 to 13.19.

[New, Appendix B] We have also substantially expanded the evaluation across architectures:

  • Recurrent backbones (Fig. 7; App. B.1.2, B.2.2): The Lattice improves Gated DeltaNet, Mamba-2, Triadic GDN and GDN–Mamba-2 hybrids at every tested state size, at approximately matched parameters. A 64K-state Lattice GDN (16.05 PPL) even outperforms a pure GDN with a 4× larger state (16.11). A spectral analysis of GDN's memory (Table 8) shows that the Lattice helps the model use its native memory more effectively.
  • Residual-stream enhancements (Fig. 5; App. B.1.1, B.2.1): We compare against hyper-connections (HC), mHC, value residuals and input residuals (x0). The Lattice alone outperforms HC, mHC and value residuals, and composes with these approaches: Lattice + HC + x0 achieves 13.61 PPL, compared with 14.22 for HC + x0 without the Lattice.

Together, these experiments show consistent benefits across training budgets, model configurations and sequence-mixing architectures. Scaling to multi-billion-parameter models remains an important direction for future work.

W3: Source code


We will publicly release the full codebase, training configurations and model checkpoints upon acceptance.

We hope this answers any of the reviewer's remaining concerns, happy to discuss further.
Best Regards.

I keep positive score,

Reviewer_4kFbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adds a "horizontal latent" to each layer of a middle "Lattice" block of a decoder-only Transformer (20 of the 22 layers in the headline models). It is a per-layer state carried over time with a Mamba-2-style gated diagonal recurrence, so training stays parallel through a scan. In each Lattice layer the state FiLM-modulates Q/K/V and shifts the gate and up pre-activations of the MLP, and the top Lattice layer's state also feeds the output head. Optional extras are input injection, a "vstream" and a Jacobi refiner (J3) that passes top-layer information back to the bottom over three sweeps.

LLaMA-style models with 400–600M parameters are trained on 10B/20B/30B FineWeb-Edu tokens. At 20B tokens, a 406M model lowers perplexity from 15.41 to 14.40 against a parameter- and FLOP-matched 24-layer Transformer, which matches a 609M, 40-layer Transformer with more layers and a larger KV cache. The paper also reports downstream gains (up to +2.2 points zero-shot average over nine tasks), large gains after GSM8K-Aug fine-tuning, and complementarity with pause tokens and PonderLM-2.

Critical strengths
  1. The module is simple and zero-initialized, so at initialization the model is exactly a plain Transformer, and the single-pass variant keeps token-parallel training. Over the same-size Transformer, the gains are consistent across three token budgets, in- and out-of-domain perplexity, and on average across nine downstream tasks. Depth- and width-scaled baselines are included. If the gains survive the controls below, this is an easy way to get more out of a fixed parameter and KV budget.
Critical weaknesses
  1. The comparison that would test the paper's main claim is missing. The horizontal latent is an input-dependent linear recurrence over time that the same layer reads. It therefore carries past-token information into the current token's Q/K/V and MLP. That is a form of sequence mixing, and the obvious alternatives are hybrid Transformer–SSM layers or short convolutions. The paper argues that its latent plays a different role from the SSM state in hybrids (Abstract, Section 2). However, it includes none of the following:

    • a hybrid baseline at matched parameters (e.g., interleaved Mamba-2/Gated DeltaNet layers, or parallel SSM heads as in Hymba, which the paper cites);
    • a short-convolution baseline (e.g., Canon layers at the same insertion points);
    • an ablation that keeps the extra parameters but removes the carry over time (the smaller "no-carry" Transformer in Section 4.3 is not a matched ablation).

    Without these, I cannot tell whether the gains come from the lattice idea or simply from adding a cheap recurrent mixer to a small Transformer. Closely related designs are also not cited: the Block-Recurrent Transformer (which already frames its recurrent cell as having a "vertical" and a "horizontal" direction), the Feedback Transformer and Mega.

  2. Several headline claims are weaker than stated.

    • The advantage shrinks with more training. Take the best single-pass 400M LLLM at each budget. Against the same-size Transformer the gain is −7.6% perplexity at 10B (15.56 vs 16.84), −6.6% at 20B (14.40 vs 15.41) and −5.4% at 30B (13.92 vs 14.72). The 600M (1024, 1024) model shows the same trend against Base-40L (−6.2%, −4.8%, −4.4%).
    • "Matches a 1.5× larger model" does not hold at 30B. Against the 609M 40-layer model, LLLM is ahead at 10B (15.56 vs 15.76) and tied at 20B (14.400 vs 14.400), but behind at 30B on FineWeb-Edu (13.921 vs 13.802) and Pile (24.148 vs 23.478), though still ahead on LAMBADA (Table 6).
    • Table 5 does not show complementarity with pause tokens. Pause tokens improve the 40-layer Transformer by 1.68 PPL (15.76 → 14.08) but the single-pass LLLM by only 0.40 (14.78 → 14.38). With pause tokens, the Transformer is better than the single-pass LLLM, and only J3 wins (13.73).
    • J3 has no baseline at its compute. The paper is open that J3 uses 1.85 GFLOPs/token (against 1.11 for Base-40L) and that an exact sequential rollout is worse than the single-pass models (14.733). But nothing else is run at J3's compute, e.g. a looped Transformer.
    • No seeds and a single learning rate. No seeds or error bars are reported, and one learning rate (1e-3) is used for every width and depth.
    • The GSM8K results are hard to trust without a protocol. None is given. Two models with the same pretraining perplexity and the same zero-shot average (Base-40L and the (908, 256) inj+vs LLLM: 14.400 and 47.36 vs 47.38; the latter is labelled re+vs in Table 4) differ by 10 points on GSM8K (33.5 vs 43.6).
Questions for authors
  1. At 400M/10B, please add matched-parameter runs for:
    • (a) a hybrid with Mamba-2/GDN layers or parallel SSM heads;
    • (b) short causal convolutions at the same insertion points;
    • (c) the Lattice with the carry over time switched off.
  2. What is the GSM8K-Aug fine-tuning protocol: CoT or not, epochs, learning rate, decoding? Was the learning rate tuned per model?
  3. Section 4.1 mentions four token-budget scales but lists three, and A.5 lists a 50B shard. Are there 50B results? Which token budgets do Tables 2 and 5 use? Also, from the architecture described in the paper I get about 643M (dv=1620) and 585M (dv=1472) parameters for the width-scaled Base-24L rows, not 603M and 606M. Which is correct?
  4. Please add a looped-Transformer baseline at J3's FLOPs, and report J3's decoding latency.
2 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer 4kFb (1/2)

We thank the reviewer for the constructive review and for recognizing the simplicity, token-parallel training, and consistent gains of our approach. The central question is: are the gains specific to the Lattice, or could they be obtained by adding a cheap sequence mixer? We added experiments on recurrent carry, short convolutions, and recurrent mixers (Appendix B).

W1 / Q1: Is the Lattice just an additional sequence mixer?


We agree that horizontal recurrence performs sequence mixing in a broad sense. Our distinction concerns how this state is used: rather than replacing the backbone's sequence mixer (KV cache or KV state), the Lattice maintains an auxiliary recurrent state that conditions its existing memory reads, writes, and computation.

Unless noted, experiments use ~400M parameters and 10B FineWeb-Edu tokens pretraining (Transformer: 16.94 PPL).

Q1(c) Is recurrent carry necessary? (Fig. 6, App. B.2.1)


We test the proposed no-carry alternative. We preserve the same E1/L20/D1 layout and zero-initialized FiLM heads on QKV and the MLP, but replace the recurrent latent with a stateless projection of either input token embedding or incoming residual stream. All variants are parameter-matched.

ConditioningQKV PPLQKV + MLP PPL
Stateless, residual stream16.6216.69
Stateless, token embedding16.2916.32
Lattice (recurrent)15.9915.63

Stateless modulation helps, but recurrent conditioning improves substantially further. With QKV+MLP modulation, removing recurrence increases PPL from 15.63 to 16.32 even with the best stateless signal. MLP modulation helps only with recurrent conditioning.

Q1(b) Short causal convolutions


We also compare short convolutions on QKV inputs, which mix recent tokens without a separate recurrent latent.

A width-4 static convolution, as in Canon-style layers (Physics of LLM 4.1 paper), reduces Transformer PPL from 16.94 to 15.87, adding only 0.28M parameters. A stronger input-dependent dynamic convolution (concurrent work, arxiv.org/pdf/2606.03825) reaches 15.52, while the Lattice reaches 15.49, with 6.8M fewer parameters than the dynamic-convolution model.

The distinction is in the temporal receptive field: width-4 convolutions directly aggregate a fixed window of recent tokens, whereas the Lattice's gated recurrence can carry a compressed summary across the full preceding sequence. With further advancements in non-linear parallelizable recurrent update rules, Lattice performance should only improve.

More importantly, the approaches are complementary: combining dynamic convolution with the Lattice further reduces PPL to 14.99. We can further even lattice-condition the input-dependent controls generating the dynamic conv kernel -- future work. This suggests that local convolutional mixing and persistent horizontal-state conditioning provide distinct, useful contributions.

Q1(a) Recurrent mixers and hybrids (Please see Fig. 7, App. B.1.2/B.2.2)


We further test whether the Lattice remains useful when the backbone already possesses a large recurrent memory. We train parameter-matched Gated DeltaNet (GDN), Mamba-2, Triadic GDN, and GDN–Mamba-2 hybrid backbones, both with and without the Lattice, across native-state sizes of 16K–256K.

Adding the Lattice lowers perplexity in every tested configuration (See Fig. 7 Appendix B). For example, 64K-state Lattice GDN (16.05) outperforms pure GDN with 4×4\times larger native memory (16.11). Thus, simply enlarging the backbone's recurrent memory does not reproduce the benefit of horizontal-state conditioning.

To understand this, we examine GDN's existing key–value memory matrix (Table 8). With the Lattice, its singular-value spectrum becomes more concentrated, while query alignment with the leading memory direction increases. These measurements are consistent with the Lattice helping the backbone organize and access its existing memory more effectively, beyond adding storage.

Furthermore we would like to share a complementary view. Lattice can be seen as a memory-aware residual stream enhancement. Towards this, we additionally compare and find that the Lattice composes with residual-stream enhancements (Fig. 5, App. B.2.1): at 400M/20B token pre-training, Lattice + HC + x0 reaches 13.61 PPL, versus 14.22 for HC + x0 alone. HC: Hyper-Connections, X0: input residual.

Related work


We will incorporate the missing citations in the related work section. Together, the no-carry ablation, convolution comparisons, recurrent-backbone results and comparisons with prior residual stream enhancements support this architectural choice: the Lattice benefits from persistent history, complements local sequence mixing, and improves models already equipped with substantial recurrent memory.

We hope this clarifies reviewer's major concerns.

AuthorsOpenReview ↗

Rebuttal Response to Reviewer 4kFb (2/2)

W2: Headline claims are weaker than stated


We thank the reviewer for the careful reading & will revise the claims as mentioned below.

Gains with more training


We agree relative gains decrease w/ longer training, as is common for arch. improvements in overtrained regimes. However, it remains clearly positive at every budget:

10B20B30B
Base-24L (406M)16.8415.4114.72
LLLM (406M)15.5614.4013.92
Base-40L (609M)15.7614.4013.80
LLLM(1024,1024) (603M)14.7813.7013.19

At 600M, Lattice beats depth-scaled Base-40L at every budget, including 30B (13.19 vs. 13.80). We agree that 406M Lattice outperforms/matches 609M Base-40L only at 10B–20B & we will restrict the claim accordingly. But even at 30B token pretraining, 406M Lattice (w/ 22 layers) is just 0.12 behind 40L 609M transformer, despite 18 fewer layers & 45% fewer KV-cache. The Lattice also composes w/ residual-stream enhancements: at 20B, Lattice + HC + x0 reaches 13.61, 11.7% below Transformer (Fig. 5, App. B.2.1). We will share 50B runs if they complete.

(Q4) A baseline at J3's compute


As an alternative to looped Transformer, we give Transformer same top-to-bottom feedback (10B tokens pre-training). Previous token's final-layer state is fed back into the vertical stream, analogous to the concurrent Full-Bandwidth Transformer & trained w/ the same 3 Jacobi sweeps. All feedback models are trained & evaluated w/ J3:

PPL
Transformer, single pass16.94
Lattice, single pass15.52
Transformer + vertical feedback (J3)15.36
Lattice + horizontal feedback (J3)15.09
Lattice + both (J3)14.92

The single-pass Lattice (15.52) nearly matches Transformer w/ vertical feedback (15.36), which reruns the full network three times. At the same sweep count (J3), feedback on the horizontal latents beats vertical feedback (15.09 vs. 15.36). Combining both is best (14.92), so the two forms of feedback are complementary.

In terms of flops, J3 runs 3 sweeps per token (~2.6× FLOPs: 1.85 vs. 0.72 GFLOP/token for single-sweep J0 runs).

Pause tokens (Table 5)


We agree and will soften this claim. We shifted compute from pause tokens to architectural studies (recurrent backbones, Fig. 7; residual-stream enhancements, Fig. 5); pause tokens need further investigation. Still, without Jacobi, the 22-layer Lattice with pause tokens (14.18) nearly matches the 40-layer Transformer with pause tokens (14.08) with 18 fewer layers (~45% less KV cache). The horizontal latent adds only a fixed-size state per layer, unlike the growing KV cache. J3 gains (13.66) also suggest that advances in nonlinear parallelizable recurrences will further boost the Lattice.

Seeds & Learning rate


  • Learning rate. We tuned peak learning rate by grid search for both Transformer & Lattice backbones. For recurrent backbones, we follow the settings of the public fla/flame repositories. We will list LR grid in App.
  • Seeds. Multi-seed pre-training at this scale exceeds our compute budget. Instead, we assess robustness through breadth. The Lattice improves:
    • Transformers at 10B, 20B & 30B tokens, at two model sizes;
    • GDN, Mamba-2, Triadic GDN & GDN–Mamba-2 hybrids at every state size (new Fig. 7);
    • HC, mHC, value residuals & x0 when combined with them (new Fig. 5).

Although we cannot quantify seed-to-seed variance without repeated pre-training runs, consistent improvement across model sizes, token budgets, recurrent architectures, native-state sizes provides evidence of robustness.

(Q2) GSM8K protocol


All models share same protocol. We fine-tune on GSM8K-Aug (385620 examples) with explicit chain-of-thought: questions are masked from loss, and reasoning steps and final answer are supervised. Training runs for 4 epochs (12K steps) with batch size 128, weight decay 0.01, and 3% warmup followed by a constant learning rate. Learning rate is tuned per model over {1e-4, 4e-4, 8e-4}; 8e-4 is best for all models. Decoding is greedy.

While pre-training perplexity & zero-shot accuracy do not fully determine fine-tuning transfer, every Lattice model is at least 10.1 points ahead of every Transformer at both 400M and 600M. We will add the protocol to the appendix and correct the "re+vs" label to "inj+vs".

Q3: Token budgets & parameter counts


  • Budgets. Overall we report "three scales" (10B/20B/30B). Most results are at 20B scale, unless otherwise reported. The ablation Figure 4, Table 5 and Table 2 use 10B tokens pretraining. We will state budgets in all captions.
  • Parameters. The counts (603.3M, 606.2M) are correct, but the "24L" labels are wrong: both width-scaled baselines have 22 layers, and the dv=1472d_v{=}1472 model uses 23 heads (attention width 1472). We will fix the labels and list the configurations.

Best Regards.

Reviewer_LBk2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces latent lattice language models. A latent lattice layer leverages an additional latent state, rtlr_t^l, with Mamba-like updates over the “input” htl−1h_t^{l-1}. This additional state affects the computation of the transformer block by modulating the query, key, value and MLP weights at each time step. Additionally, one can inject token embeddings or high-level representations of past tokens into rtlr_t^l to improve performance (at the cost of additional FLOPs). These models are evaluated by perplexity and accuracy on many commonsense reasoning tasks with 400-600M parameter models.

Critical strengths
  • The project page offers a great visual explanation, alongside all the method diagrams.
  • The general idea of leveraging past information from deeper layers is an exciting area of research.
Critical weaknesses

I do not think the base Latent Lattice layer bypasses the stated limitations of transformers, making the motivation behind this layer ill-posed. I also do not buy the author’s argument that transformers are bottlenecked because they use one latent instead of two; this is a semantics argument.

  • The authors claim that transformers are bottlenecked because they use a single latent to read and write to memory. They further point out that the KV cache is a persistent read-only record, “written once and never subsequently revised” [line 191]. I would argue that the KV Cache’s append-only updates is what makes it so powerful, as past information is not overwritten. At each time step, the query has all the available information to make decisions. Furthermore, one can argue that the query, key, and value are different latents that have different roles.
  • Besides the jacobi iteration variant, r_t offers a very similar form of memory to the KV cache. This is just applied in a different way with modulation.

While the empirical results show mild gains above transformers in terms of perplexity and accuracy on commonsense reasoning tasks, these are not significant or robust enough to compensate for the increase in layer design complexity.

Questions for authors

The biggest overall question is “Why this specific implementation?”. A simplified explanation of latent lattice layers is that they use a state-dependent modulation of query, key, value and MLP weights. Therefore, how important is r_t^l’s update rule (Mamba2) compared to using other SSM / linear attention methods? How does this compare to using input-dependent, but still state-agnostic, modulation parameters (i.e. instead of depending on some state r_t^l, use an alternative function of h_t^l and/or e_t)?

Additionally, should we, as LLM researchers, adopt latent lattice layers over traditional transformer blocks in all settings, or are we making a tradeoff somewhere?

How do you define or measure “computational latent capacity” per line 19?

2 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer LBk2 (1/2)

We thank the reviewer for the careful assessment and appreciation of our project page, diagrams, and research direction. We have clarified the motivation and added experiments addressing these concerns (Appendix B of the revised PDF).

We request the reviewer to look into the interesting result figures & table in Appendix B, which demonstrate the Lattice's effectiveness across recurrent backbones and its complementarity with existing residual-stream enhancements. Results also uploaded on webpage: https://latent-lattice-language-model.netlify.app/

W1: KV cache and computational capacity


We agree that the append-only KV cache is a strength of Transformers. The Lattice preserves this mechanism: no entries are overwritten, attention retains access to the same history.

Our argument is not that Q, K, and V lack distinct roles, but that they are projections of the same residual-stream representation. We ask whether computation benefits from additional per-layer states that persist across tokens. These horizontal latents evolve recurrently and condition QKV and MLP computations, influencing how memory is read and written.

This complements residual-stream enhancements such as hyper-connections (HC), mHC, and value residuals, which improve information flow across depth. The Lattice instead introduces persistent computation across time. We now compare against these methods (Fig. 5, App. B.2.1). The Lattice alone achieves 14.47 PPL, versus HC (15.28), mHC (14.93), and value residuals (14.69). More importantly, Lattice + HC + x0 reaches 13.61, versus 14.22 for HC + x0 without the Lattice.

We agree that original "one latent vs. two" framing could appear semantic. Our revised motivation will emphasize the architectural distinction: persistent computational states across time that complement existing residual pathways.

W2: Is the horizontal latent just another memory?


We ran multiple experiments to address this.

Both carry historical information, but serve different roles. The KV cache stores token-wise entries for attention retrieval and grows with sequence length. The horizontal latent is a fixed-size state updated every token, which conditions memory reads/writes and MLP computation rather than storing individually retrievable entries.

We test this distinction by incorporating the Lattice into GDN, Mamba-2, Triadic GDN, and GDN–Mamba-2 hybrids (Fig. 7, App. B.1.2/B.2.2). These backbones already maintain substantial recurrent memories; the Lattice preserves their native memory mechanisms while conditioning their operations. At approximately matched parameters, perplexity improves at every tested native-state size. Notably, 64K-state Lattice GDN (16.05) outperforms 256K-state pure GDN (16.11), despite having 4× fewer native-memory entries -- thus Lattice is not about more memory, rather better written memory.

We further examine how the existing memory changes (Table 8). Each GDN head maintains a recurrent key–value memory matrix. We analyze its singular values: lower effective and stable ranks indicate that memory is concentrated in fewer dominant directions. We also measure how strongly retrieval queries align with the leading memory direction.

With the Lattice, GDN's memory becomes more concentrated (effective rank 37.92 → 33.42 at 256K), while query alignment increases (7.96 → 13.32). These findings support our hypothesis that horizontal latents help the backbone organize and access its existing memory more effectively, rather than merely increasing storage.

W3: Gains versus architectural complexity


The improvements are substantial across scaling, longer training, and downstream transfer:

  • Scaling efficiency: At 20B tokens, a 406M Lattice matches a 609M Transformer (14.40 PPL), using 33% fewer parameters, 35% fewer estimated inference FLOPs, and a 1.8× smaller KV cache.
  • Longer training: At 30B tokens, FineWeb-Edu PPL improves by 5.4% at 400M and 4.4% at 600M.
  • Downstream transfer: At 400M, nine-task average accuracy improves from 45.46 to 47.67 zero-shot and 46.19 to 47.79 five-shot. After GSM8K finetuning, the matched models achieve 44.5% versus 29.7%.
  • Architectural generality: Gains also extend across the recurrent backbones and residual-stream enhancements discussed above.

The base Lattice adds a diagonal affine recurrence and FiLM conditioning per layer, preserving token-parallel training through an associative scan. We offset its parameter cost by narrowing the residual stream, keeping comparisons approximately matched in parameters and estimated FLOPs.

The optional Jacobi refiner training incurs additional compute (~2.6× FLOPs for 3 sweeps). During autoregressive inference, its feedback can instead be computed exactly in a single sequential pass at base FLOPs. The main single-pass Lattice results are without costly Jacobi refinement.

AuthorsOpenReview ↗

Rebuttal Response to Reviewer LBk2 (2/2)

Q1: Would input-dependent, stateless modulation suffice?


We thank the reviewer for a great suggestion. We now directly test this alternative (Fig. 6, App. B.2.1). Our architecture uses recurrent horizontal latents to condition QKV and MLP computations through FiLM. But could the gains come simply from additional conditioning, without recurrence?

We replace the recurrent latent with a stateless signal derived from either the token embedding or the incoming residual stream, retaining the same zero-initialized FiLM heads. All models have ~406M parameters and are trained on 10B tokens.

Conditioning signalQKV PPLQKV + MLP PPL
Transformer (no modulation)16.9416.94
Stateless, residual stream16.6216.69
Stateless, token embedding16.2916.32
Lattice (recurrent)15.9915.63

Three observations:

  1. Stateless FiLM helps (16.94 → 16.29), confirming that additional conditioning is useful.
  2. Recurrence improves further (16.29 → 15.99 for QKV; 16.32 → 15.63 for QKV+MLP), at matched parameters.
  3. MLP modulation helps the recurrent Lattice (15.99 → 15.63), but not stateless conditioning (16.29 → 16.32).

Thus, the gains cannot be explained by additional FiLM conditioning alone. The history accumulated in the horizontal latent provides a substantial further benefit.

We chose the Mamba-style gated diagonal affine recurrence because it combines input-dependent updates with an exact associative scan, preserving token-parallel training. Our contribution is the role of the recurrent state in conditioning computation, rather than a new recurrent operator. We do not claim this update is optimal; richer updates, including delta-rule variants, are promising future work.

Q2: Should the Lattice replace Transformers universally?


We view the Lattice as a complementary architectural enhancement, not a universal replacement. It improves Transformers, recurrent models, and hybrids across our tested settings.

Tradeoffs to ensure iso-parameter comparison: Our principal comparisons use approximately matched parameters and estimated FLOPs. For the base Lattice, we offset the additional recurrent-state and FiLM parameters by slightly narrowing the residual stream (e.g., 1024 → 945). The base recurrence remains token-parallel through an associative scan. While theoretical FLOPs are comparable, actual runtime depends on kernel efficiency; optimizing the implementation is future work.

The optional Jacobi refiner is the main computational tradeoff. Its nonlinear top-to-bottom feedback requires three training/evaluation sweeps (~2.6× forward FLOPs). During autoregressive inference, this feedback can instead be computed exactly in a single sequential rollout at base FLOPs, although its predictions may differ from three-sweep evaluation.

Importantly, the main single-pass Lattice achieves its improvements without Jacobi refinement.

Q3: What is computational latent capacity?


We define computational latent capacity as a measure of the latent states participating in computation, rather than a formal information-theoretic bound.

  • Transformer: A single vertical residual stream of width dmodeld_{model} propagates through depth.
  • Lattice: Alongside this stream, each of the LL Lattice layers maintains an independently evolving horizontal latent of width drd_r, giving dmodel+Ldrd_{model} + Ld_r total state dimensions across the architecture.

These horizontal states persist across tokens and modulate the computations of the vertical stream. We agree with the reviewer sentiment that more than the capacity, its the role of the horizontal latents which lead to performance improvements. This dimension count characterizes the architecture's state allocation, not its parameter count or FLOPs.

The allocation matters empirically: a 291M-parameter Lattice(512,512) nearly matches a 406M Transformer (16.96 vs. 16.94 PPL), demonstrating that distributing capacity across vertical and horizontal pathways can be more parameter-efficient than widening the residual stream alone.

We hope this answers the reviewer's concerns, happy to discuss further. Best Regards.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response — New Results in Revised Manuscript (Highlighted in Red)

General Response to AC and Reviewers

Dear AC and Reviewers,

We thank all reviewers for their careful reading and constructive feedback. We appreciate their recognition of the novel RNN–Transformer integration (ddsB), promising research direction (LBk2), and simple, zero-initialized, token-parallel design with consistent gains (4kFb).

In response to the feedback, we conducted new experiments, presented in Appendix B (highlighted in red) of the revised manuscript. Please visit this anonymous webpage for method animations and results: https://latent-lattice-language-model.netlify.app/

Key findings include:

  • Residual-stream enhancements (Fig. 5). On its own, Lattice can be seen as a memory-aware residual stream enhancement approach. The Lattice outperforms HC, mHC, and value residuals at matched parameters, while also composing with them. Lattice + HC + x0 achieves 13.61 PPL, versus 14.22 for HC + x0 (400M, 20B tokens).
  • Is recurrent history necessary? (Fig. 6). Replacing the recurrent latent with stateless conditioning while retaining the same FiLM heads and parameter budget confirms the importance of history: 15.63 vs. 16.32 PPL for recurrent versus the best stateless modulation.
  • Recurrent backbones (Fig. 7). The Lattice improves GDN, Mamba-2, Triadic GDN, and GDN–Mamba-2 hybrids across all tested native-state sizes (16K–256K). Notably, 64K-state Lattice GDN (16.05) even matches pure GDN with 4×4\times larger native memory (16.11) -- highlighting Lattice is more about better writing the memory than just more memory.
  • Memory geometry (Table 8). The Lattice concentrates GDN's memory spectrum and increases query alignment with its leading memory direction (7.96 → 13.32 at 256K), consistent with more effective use of existing memory.

Our reviewer-specific responses further present short and dynamic convolution comparisons (reaching 14.99 PPL when combined with the Lattice), a Jacobi control with vertical feedback, the GSM8K fine-tuning protocol, and clarifications and corrections to the original claims.

We are encouraged by these new results and hope they address the reviewers' central concerns. We look forward to further discussion.
Best Regards.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.