- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
I provide the review regarding “Latent Lattice Language Models”. This study provides a novel method to incorporate an RNN-like state and Transformer-like KV cache. The proposed method passes through the latent space along with time and incorporates a KV cache in each layer. The proposed method achieves better performance with fewer parameters and FLOPs than Transformers. The authors also provide detailed ablation test that the proposed components contribute to the performance gain.
Critical strengths
This is a very impressive study, especially in the experimental results. The authors provide a novel method to incorporate RNN and Transformer mechanisms and achieve better performance than Transformers. This is very impressive because generally RNNs and Transformer variants are faster but have lower performance. Transformers are the best method, but they are computationally expensive. The experimental results suggest that the authors' method has better performance with lower computational cost. This is very important because in the past 10 years, Transformers have been the upper bound of neural network architectures, and others have only advantages in task-specific or computational aspects. Figure 4 is very impressive and shows that the proposed components are useful. I feel something great breakthrough in this study.
Critical weaknesses
I find several Weaknesses in this study.
-
The authors provide apps and a detailed explanation in the main text and appendix. I believe the authors' effort is extensive, and I do not think this is a major fault, but I still cannot understand the details and the whole picture of the proposed method. I can understand each component; however, understanding the connection between each component and the procedure of training and inference is still difficult to follow. Additional figures, videos, and mathematical formulas may help readers.
-
The baseline PPL of the experiments is not small enough. They are around 10, which is marginally acceptable at an academic level; however, if the authors have the computational resources, I hope they can run experiments on larger LLMs with larger datasets.
-
In addition, the source code has not been released.
Questions for authors
The proposed method seems to inject RNN latent space into the KV cache, but I could not understand the whole mechanism despite the authors' effort not only main-text but also apps. It seems that the nature of the proposed method is a complicated mechanism, and existing way to express it cannot capture it. I hope the authors provide more figures and mathematical formulas, especially to connect all components for better understanding. I can understand each component and some intuition, but it is still difficult to understand, and I cannot give clear acceptance. However, I provide weak acceptance score because the experimental results are impressive.
2 comments
I keep positive score,
Rebuttal Response to Reviewer ddsB
We thank the reviewer for their encouraging assessment of our contribution and experimental results. We agree that the original presentation could better connect the components and clarify training and inference. We have reorganized Section 3 and improved the accompanying explanations and figures (Figs. 1–3).
We would also like to highlight the new results in Appendix B (shown in red in the revised manuscript), demonstrating the Lattice's effectiveness across recurrent backbones and its complementarity with existing residual-stream enhancements.
W1 / Q1: Understanding the components and training/inference procedure
Core mechanism (Sec. 3.2, Fig. 1). The Lattice adds a horizontal recurrent latent rtℓ at each Lattice layer. It persists across tokens and assists the vertical residual stream htℓ flowing through depth. Each layer performs two operations:
We study three enhancements to the horizontal pathway (Sec. 3.2.2), each ablated in Fig. 4 (Sec. 4.3):
Training. The recurrence coefficients depend only on current-token inputs, not on the previous horizontal state. An associative parallel scan therefore computes all horizontal states at each layer across tokens, preserving token-parallel teacher forcing. Input injection and vstream retain this property. The optional non-linear top-to-bottom feedback instead uses three token-parallel Jacobi sweeps (J3), at additional compute cost.
Inference. Generation is autoregressive. At each Lattice layer, we update the horizontal state, use it to condition attention and the MLP, and advance the vertical stream. Each layer maintains one fixed-size recurrent state per sequence and adds no KV-cache entries. In sequential rollout, the optional top-to-bottom feedback is evaluated exactly, without Jacobi sweeps. We report rollout and J3 evaluation separately.
W2: Larger models and datasets
We agree that validation at larger scales is important. Our compute budget currently limits experiments to sub-billion-parameter models. Within that budget, we provide evidence across 10B, 20B and 30B training tokens, including extended post-Chinchilla pretraining:
[New, Appendix B] We have also substantially expanded the evaluation across architectures:
Together, these experiments show consistent benefits across training budgets, model configurations and sequence-mixing architectures. Scaling to multi-billion-parameter models remains an important direction for future work.
W3: Source code
We will publicly release the full codebase, training configurations and model checkpoints upon acceptance.
We hope this answers any of the reviewer's remaining concerns, happy to discuss further.
Best Regards.