acceptodds
Under review as a conference paper at ICLR 2027

Internal Attention: Growing Token Representations Without Growing the FFN or the KV Cache

Abstract

Increasing the width of a token's representation is expensive: parameters grow quadratically, dominated by the feed-forward network (FFN), and the KV cache grows linearly. We combine two ideas into an architecture that avoids both costs. First, inspired by Hyper-Connections, we split the token representation into a compute subspace and a memory subspace, where the FFN block is applied to the compute subspace only. Following Multi-Head Latent Attention, we also restrict the attention's key and value projections to the compute width. Quadratic costs in the number of parameters are therefore tied to the compute width and decoupled from the memory width. Second, we introduce a mechanism different from Hyper-Connections for moving information between the memory and compute subspaces: an attention internal to the token representation, which separates it into a fixed number of internal tokens and applies attention among them rather than across the sequence. An internal attention on the full vector followed by an FFN on the compute part forms a block that can be stacked, increasing depth without increasing the KV cache. Experiments on small autoregressive language models and integer multiplications show that the resulting architecture improves on both standard transformers and Manifold-Constrained Hyper-Connections (mHC) at equal KV cache, including in parameter efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Deep Learning, Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FmvwOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This manuscript studies how to increase a Transformer’s representational capacity without proportionally increasing FFN cost or KV-cache size. It proposes the Internal Attention Transformer (IAT), which splits each token representation into compute and memory components, applies the FFN only to the compute part, and uses an internal attention mechanism to exchange information between them. This allows the model to add memory capacity and internal depth without increasing the KV cache. Experiments on TinyStories and integer multiplication show that IAT outperforms standard Transformers and mHC at comparable KV-cache sizes, with better validation performance, parameter efficiency, and arithmetic accuracy.

Critical strengths

1.) The proposed internal attention mechanism is simple and straightforward (in a good way). It operates entirely within each token representation, preserves autoregressive causality, and can be stacked to increase model capacity without increasing the KV-cache size.

2.) The manuscript also includes useful controlled comparisons against both standard Transformers and mHC, together with parameter-efficiency sweeps that vary key architectural choices. These experiments help show that the reported gains are not solely due to using a larger number of parameters.

Critical weaknesses

1.) The manuscript is difficult to follow due to fragmented exposition, abrupt transitions between ideas, and a lack of a clear overarching narrative. These presentation issues significantly hinder readability and, in my view, the manuscript would require substantial restructuring before it is ready for acceptance.

2.) The evaluation is very limited in scale, covering only TinyStories and a synthetic 2-3 digit multiplication task with very small models. As a result, it is unclear whether the proposed compute/memory separation would remain effective for realistic modern language models and larger-scale inference settings.

3.) The manuscript is motivated by KV-cache efficiency, yet it provides no real systems-level evaluation. While the proposed architecture keeps KV-cache size fixed, it introduces additional computation through internal attention and wider representations; therefore, measurements of latency, throughput, memory usage, or FLOPs are needed to demonstrate an actual efficiency benefit.

4.) The baseline coverage is limited, with experiments focusing mainly on a standard Transformer and mHC. Given that the proposed method targets memory/computation decoupling and KV-cache efficiency, comparisons against a broader set of architectures with similar goals would be needed to establish a stronger empirical advantage. Several relevant efficient sequence-modeling approaches, including Performer [1], CosFormer [2], Reformer [3], RACE Attention [4], Mamba [5], and recent hybrid architectures such as Qwen3.5, are not discussed. Including this literature would help better position the proposed method and clarify why the evaluation focuses primarily on vanilla Transformers when modern architectures are increasingly moving toward more efficient or hybrid designs.

5.) The experiments do not directly validate that the added “memory” dimensions actually function as memory. The evidence is mainly improved task performance, without probing or ablations showing what information is stored, how it is used, or whether the gains simply come from increased architectural capacity.

[1] Choromanski, K., Likhosherstov, V., et al. Rethinking Attention with Performers. ICLR 2021.

[2] Qin, Z., Sun, W., et al. cosFormer: Rethinking Softmax In Attention. ICLR 2022.

[3] Kitaev, N., Kaiser, L., et al. Reformer: The Efficient Transformer. ICLR 2020.

[4] Joshi, S., Chowdhury, A., et al. RACE Attention: A Strictly Linear-Time Attention Layer for Training on Outrageously Large Contexts. ICLR 2026.

[5] Lahoti, A., Li, K., et al. Mamba-3: Improved Sequence Modeling using State Space Principles. ICLR 2026.

Questions for authors

1.) Refer to Weaknesses.

2.) The multiplication experiments show substantial variance across seeds and learning rates, and the manuscript reports the best-performing run for each architecture. Could the authors instead report mean and variance across runs, and explain whether internal attention is consistently easier or harder to optimize than the baselines?

3.) Since internal attention is bidirectional within each token representation, could the authors provide a more explicit explanation or formal argument for why this cannot leak information across sequence positions?

Reviewer_zEGEOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Internal Attention (IAT) provides a mechanism to scale the memory size of a tokens' embedding independent of the compute size, the size used in FFN and Attn operations (similar to mHC). The connection from the memory part to the compute part is established through "internal attention", which applies attention across the sub-embeddings of the total embedding. On a tiny, uncommon benchmark (Tinystories PPL), it outperforms standard transformers and the similar in spirit Manifold-Constrained Hyper-Connections. Scaling behavior remains to be shown.

Critical strengths

Scaling up activations is close to free in decoding, mildly expensive in parallel pre-fill. At the trained scales, IAT shows largely better performance than mHC at roughly equal cost and parameters.

Critical weaknesses

Scaling up activations leads to very heavy memory limitations in training. It is unclear how this behaves on small scales and above (training on a few million parameters on tiny stories). A scaling analysis would be desirable, also performance on downstream evaluations. Using the temporal attention on the "compute" part of the residual only leads to worse performance than for the vanilla transformer. Probably parameters are not matched here, so it hints at a worse architecture. As model size is typically a very strong performance indicator, additional parameters on one side (low-rank projection) should always be compensated (e.g. MLP size adaptation).

Questions for authors

Can you train this on a reasonable size? Or provide a scaling analysis testing a few different scales for each approach. Can you do a qualitative analysis on "switching behavior" and "dead dimension" (so memory that is never attended to / with low attention score)?

Reviewer_SYg5OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Increasing the width of token representations in standard Transformers improves model capacity, but causes feed-forward network (FFN) parameters to grow quadratically and the KV cache to grow linearly. To avoid both costs, the authors propose the Internal Attention Transformer (IAT), which splits each token representation into a compute component (xcx_c) and a memory component (xmx_m). While the FFN operates only on xcx_c and temporal attention projects keys and values from the full representation to the compute width dd, an Internal Attention mechanism moves information between the compute and memory subspaces by splitting the full vector into kk internal tokens and applying bidirectional attention among them. Experiments on small-scale language modeling (TinyStories V2) and 2-to-3-digit integer multiplication show that IAT outperforms standard Transformers and Manifold-Constrained Hyper-Connections (mHC) under an equal KV-cache budget (Tables 2 and 3).

Critical strengths
  • Well-Motivated Problem. The paper highlights an important scaling trade-off in Transformer architectures through an intuitive CPU register-and-memory analogy (Lines 039–046). Decoupling the token representation width from both the quadratic FFN parameter growth and the linear KV cache growth is a clear and practical research direction.

  • Clever Reframing and Adaptation of mHC as a Memory Mechanism. As discussed in Lines 057–064, the Hyper-Connections line of work was introduced to extend residual connections and address the post-norm vs. pre-norm trade-off, and the authors reframe its multi-vector residual structure as adding an extra memory level. Adapting this concept and replacing the stream-level scalar mixing of mHC with content-dependent attention among internal chunks to grow token representations without expanding the FFN or KV cache is a clever architectural idea.

  • Clear Presentation. The paper is easy to follow, and both the architecture and the chunk-level internal attention mechanism are intuitively explained through clear diagrams (Figures 1 and 2) and layer-wise parameter comparisons (Table 1).

Critical weaknesses

W1. Lack of Qualitative Analysis on the Memory Component and Internal Attention. The core premise is that Internal Attention moves relevant information between xcx_c and xmx_m, and Remark 1 describes the possibility of pointer-like, address-based fetching under suitably chosen positional encodings and attention matrices. However, **the paper provides no direct analysis of learned internal attention patterns or the information encoded in $x_m$**. It therefore remains unclear how information is stored and routed, and whether the pointer-like behavior emerges in trained models.

W2. Limited Evaluation Scale and Statistical Reliability. A fixed KV-cache width could be particularly useful for long-context inference, but the experiments are limited to small models, 512-token contexts on TinyStories, and multiplication of two 2-to-3-digit integers. Although the keys and values are projected from the full representation, information from past tokens is exposed through a dd-dimensional bottleneck, and it is untested whether the gains hold on long-range retrieval tasks. In addition, the TinyStories results come from a single seed. The substantial variability shown in Figures 5 and 6 makes results across seeds and an evaluation excluding the run-selection pairs important for assessing reliability.

W3. Missing Controlled Comparisons Under Matched Parameter or Compute Budgets. Figure 4 compares validation loss against total parameter count, but does not directly compare model pairs matched in parameter count. Widening the plain Transformer also increases embedding and output-layer parameters, whereas IAT retains width dd for these components and adds capacity within the layers (Table 1). Thus, the comparison does not isolate the benefit of internal attention from the choice of where to allocate additional capacity. Moreover, sharing internal projections across chunks reduces parameter count without eliminating their repeated application or the pairwise-attention computation, and the paper provides no matched-FLOPs comparison. A more controlled baseline would keep dd and the KV cache fixed and spend the extra budget on, for example, a wider FFN, with separate comparisons under matched parameter and FLOPs budgets.

*As a minor presentation point, Figure 6 currently appears after the start of the conclusion and would be better placed near its discussion in Section 3.2.

Questions for authors

Q1. Qualitative Evidence of Memory Routing. Could you provide qualitative analyses such as internal attention weight visualizations or probing of xmx_m and xcx_c to demonstrate what kind of information is stored in the memory component and whether the address-fetching behavior described in Remark 1 actually occurs in trained models?

Q2. Long-Context Evaluation and Multi-Seed Stability. Have you evaluated whether IAT retains its advantage beyond 512 tokens, particularly on long-range retrieval tasks where information from the full past representation is exposed through dd-dimensional keys and values? Could you repeat the TinyStories experiments with multiple seeds, report the mean and standard deviation across seeds for TinyStories and for each learning rate in the multiplication experiments, and report multiplication results on validation pairs excluded from run selection?

Q3. Equal-Budget Comparison by Reallocating Capacity Within the Layers. If a standard Transformer retains the four-layer TinyStories setup with d=64d=64, and hence a KV cache of 512 scalars per token, and instead allocates the extra budget introduced by IAT to a wider FFN hidden dimension or to additional query heads with shared keys and values, how does IAT compare under matched parameter budgets and, separately, matched FLOPs budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.