- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to improve recall and long-context modeling in linear attention with little additional parameter cost. It proposes triadic linear attention, which expands the recurrent state into a third-order tensor using two keys and a value. The method supports forgetting, the delta rule, and chunkwise-parallel training.
Critical strengths
- The paper proposes a way to enlarge recurrent memory with little parameter overhead, and it substantially improves recall. The method also works with gated delta updates and efficient chunkwise training, which makes it useful for modern linear-attention models.
- The experiment setup is well controlled. At matched state size and parameter count, the method outperforms the alternatives for enlarging the state, and the gains hold for both GDN and sGLA. The upcycling and hybrid-model results further show that the method also benefits pretrained models and hybrid architectures.
- The paper also contributes efficient GPU kernels that make the larger state practical, keeping the training overhead moderate relative to the base model.
Critical weaknesses
The comparisons with alternative state-expansion strategies are limited to the 400M scale. The 1.3B results are reported for a single seed and do not include the competing expansion strategies. Although the results show improvements over the base models, the robustness of some smaller perplexity differences and the advantage over alternative expansion methods at larger scales remain less clear.
Questions for authors
Could the authors report the variance across seeds for the comparison with other ways of enlarging the state? It would also help to include at least the strongest alternative at the larger scale, to show that the advantage still holds as the model scales.
2 comments
Thanks for the additional experiments. The multi-seed results and the state-matched comparisons at 1.3B help address my main concerns. I appreciate the authors' efforts and will keep my rating at 3.
We thank the reviewer for their review. We have addressed both points below:
Seed robustness
We report the standard deviations of our 400M GDN runs across three training seeds below. Generally, perplexity evaluations are robust to training seed, while performance on recall-intensive tasks varies more. Even the worst seed for Triadic GDN achieves lower perplexities than the best seed of every state-matched baseline.
We additionally ran GDN and Triadic GDN at the 1.3B scale for a second training seed and found the results to closely match the first seed in perplexity evaluations:
Additional baseline comparisons at 1.3B scale
We had only included comparisons to state-size matched baselines at the 400M scale in the paper. We have produced additional results for grouped values (2x) and larger heads (4x) (the strongest baselines at the 400M scale) at the 1.3B scale. At this larger scale, Triadic Gated DeltaNet again outperforms the corresponding state-matched baselines across all benchmarks, for both 2x and 4x state-size increase: