- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This manuscript studies how to increase a Transformer’s representational capacity without proportionally increasing FFN cost or KV-cache size. It proposes the Internal Attention Transformer (IAT), which splits each token representation into compute and memory components, applies the FFN only to the compute part, and uses an internal attention mechanism to exchange information between them. This allows the model to add memory capacity and internal depth without increasing the KV cache. Experiments on TinyStories and integer multiplication show that IAT outperforms standard Transformers and mHC at comparable KV-cache sizes, with better validation performance, parameter efficiency, and arithmetic accuracy.
Critical strengths
1.) The proposed internal attention mechanism is simple and straightforward (in a good way). It operates entirely within each token representation, preserves autoregressive causality, and can be stacked to increase model capacity without increasing the KV-cache size.
2.) The manuscript also includes useful controlled comparisons against both standard Transformers and mHC, together with parameter-efficiency sweeps that vary key architectural choices. These experiments help show that the reported gains are not solely due to using a larger number of parameters.
Critical weaknesses
1.) The manuscript is difficult to follow due to fragmented exposition, abrupt transitions between ideas, and a lack of a clear overarching narrative. These presentation issues significantly hinder readability and, in my view, the manuscript would require substantial restructuring before it is ready for acceptance.
2.) The evaluation is very limited in scale, covering only TinyStories and a synthetic 2-3 digit multiplication task with very small models. As a result, it is unclear whether the proposed compute/memory separation would remain effective for realistic modern language models and larger-scale inference settings.
3.) The manuscript is motivated by KV-cache efficiency, yet it provides no real systems-level evaluation. While the proposed architecture keeps KV-cache size fixed, it introduces additional computation through internal attention and wider representations; therefore, measurements of latency, throughput, memory usage, or FLOPs are needed to demonstrate an actual efficiency benefit.
4.) The baseline coverage is limited, with experiments focusing mainly on a standard Transformer and mHC. Given that the proposed method targets memory/computation decoupling and KV-cache efficiency, comparisons against a broader set of architectures with similar goals would be needed to establish a stronger empirical advantage. Several relevant efficient sequence-modeling approaches, including Performer [1], CosFormer [2], Reformer [3], RACE Attention [4], Mamba [5], and recent hybrid architectures such as Qwen3.5, are not discussed. Including this literature would help better position the proposed method and clarify why the evaluation focuses primarily on vanilla Transformers when modern architectures are increasingly moving toward more efficient or hybrid designs.
5.) The experiments do not directly validate that the added “memory” dimensions actually function as memory. The evidence is mainly improved task performance, without probing or ablations showing what information is stored, how it is used, or whether the gains simply come from increased architectural capacity.
[1] Choromanski, K., Likhosherstov, V., et al. Rethinking Attention with Performers. ICLR 2021.
[2] Qin, Z., Sun, W., et al. cosFormer: Rethinking Softmax In Attention. ICLR 2022.
[3] Kitaev, N., Kaiser, L., et al. Reformer: The Efficient Transformer. ICLR 2020.
[4] Joshi, S., Chowdhury, A., et al. RACE Attention: A Strictly Linear-Time Attention Layer for Training on Outrageously Large Contexts. ICLR 2026.
[5] Lahoti, A., Li, K., et al. Mamba-3: Improved Sequence Modeling using State Space Principles. ICLR 2026.
Questions for authors
1.) Refer to Weaknesses.
2.) The multiplication experiments show substantial variance across seeds and learning rates, and the manuscript reports the best-performing run for each architecture. Could the authors instead report mean and variance across runs, and explain whether internal attention is consistently easier or harder to optimize than the baselines?
3.) Since internal attention is bidirectional within each token representation, could the authors provide a more explicit explanation or formal argument for why this cannot leak information across sequence positions?