acceptodds
Under review as a conference paper at ICLR 2027

The Extender: A Log-Structured Transformer

Abstract

We introduce the Extender, a log-structured variant of the standard Transformer architecture. In a standard Transformer, each layer communicates with subsequent layers exclusively via the residual , a superposition channel. The Extender adds a concatenation channel : each layer emits both a residual update which is added to , and a much smaller \it extension which is appended to . While both the FFN and see , the attention projections take only as input. As a result, the fully \it extended contains the complete input for the projections of all layers, reducing the persistent attention memory footprint from to . We find that with , the Extender matches Transformer accuracy on short-context (CORE) tasks at 199M-924M parameters, and exceeds Transformer accuracy on long-context (RULER) workloads, again at 924M parameters. For our 1664-wide, 924M model, the Extender's persistent attention memory footprint is smaller than MHA. The memory savings grow with model width.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, machine learning, attention, cache, memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pYuXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The Extender focuses on the persistent memory footprint: memory required between turns. It separates the superposition channel h focused on next-token prediction, from the concatenation channel x dedicated to attention memory. For every size class, 32 features of attention memory per layer was enough to match the Reference Transformer on short-context tasks, and outperform it on long-context tasks.

Critical strengths

The Extender is a log-structured variant of the standard Transformer architecture. It adds a concatenation channel x: each layer emits both a residual update which is added to h, and a much smaller extension which is appended to x. While both the FFN and q see h, the attention kv projections take only x as input. For the 1664-wide, 924M model, the Extender’s persistent attention memory footprint is 104x smaller than MHA.

Critical weaknesses
  1. The paper relies heavily on empirical observations rather than rigorous theoretical analysis to explain why separating the residual stream h and extended embedding x improves long-context performance. There is no formal proof or bound showing that the log-structured design can preserve long-range information better than a Transformer.
  2. The authors observe that the Extender outperforms a Transformer on multi-key tasks while the Transformer performs better on variable tracking tasks, which indicates different inductive biases. However, the paper provides no further investigation or mechanistic interpretation for this phenomenon.
  3. The Extender introduces KV re-projection cost during inference. The paper only qualitatively states that the overhead is amortized, but lacks detailed latency / throughput measurement on real hardware (GPU). Readers cannot see the trade-off: how much extra compute cost must be paid to achieve the 104× memory reduction.
  4. The empirical evaluation is incomplete: the authors only conduct comparisons with a Transformer model, and omit benchmarking against other competing algorithms, particularly existing SOTA approaches for long-context memory reduction.
Questions for authors

The work suffers from limited theoretical grounding, narrow scaling experiments, incomplete benchmark evaluation, and insufficient measurement of inference compute-memory tradeoffs. Several interesting inductive bias phenomena are observed but left unexplained.

Reviewer_FVtkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Extender, a Transformer variant in which two channels connect the layers instead of one: a standard residual and a second one, to which each layer appends a small number of features. Attention keys and values only read from the second channel. The authors report a 104x smaller persistent attention footprint than MHA for their 920M model. At up to 924M parameters, the Extender matches a Llama-style Transformer on DCLM CORE and after long-context training, it outperforms the Transformer on RULER up to 64k context.

Critical strengths
  1. The Extender reduces the persistent attention memory footprint by a factor of 104 compared to a MHA Transformer at matched parameters count, while also matching its accuracy on short-context CORE tasks.

  2. On RULER, more specifically, on MK2 and MK3, the Extender outperforms the parameter-matched Transformer.

  3. The authors investigate several architectural decisions, including extension dimensionality, attention inputs, and window selection. The additional experiments with GQA and knowledge distillation are also valuable.

Critical weaknesses
  1. The 104x reduction is measured against MHA only, no GQA or MLA Transformer baselines are presented. All memory numbers are analytical, with no measured memory, latency, or throughput.

  2. From 2.5B to 5B ProLong tokens, Extender’s score on RULER lead drops at all lengths up to 32k (but rises at 64k), so long-context advantage doesn’t grow steadily with more long-context training.. Experiments also stop at 924M parameters with Chinchilla budgets. It remains unclear whether the benefits extrapolate to larger model sizes and longer training regimes.

  3. All models are trained with a single seed, and RULER uses just 30 samples per task. The default 199M Extender reports a CORE of 14.6 in Table 4 but 15.7 in Table 8.

  4. Analysis of what the extension channel actually contains is limited. The paper suggests that layers may need a separate channel for features like token distances, but doesn’t provide evidence for this claim. The Transformer’s advantage on variable tracking is also reported but not investigated.

I’m leaning towards rejection because of weaknesses 1 and 2. A GQA Transformer baseline and a longer ProLong comparison could change my assessment.

Questions for authors
  1. How are DCLM CORE scores centered? What does zero represent in Figure 2?

  2. Could the authors provide the cost of rematerializing the KV cache from x* at the start of a turn, compared to re-prefilling a Transformer?

  3. Both models use the same RoPE hyperparameters, yet before ProLong training the Transformer collapses beyond 2k while the Extender degrades gradually. What could explain this?

  4. What explains the performance jump of Extender on SQUAD at 924M? After ProLong training the gap seems to disappear.

Reviewer_j9weOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes the Extender, a transformer architecture where each layer appends a small set of features to a shared representation used to compute keys and values. This allows the model to retain a compact record between conversation turns and reconstruct the full KV cache when needed, without rerunning the full model on previous tokens.

Critical strengths

The idea and the results look interesting. Proposed method reduces persistent KV cache memory while achieving comparable performance on short-context tasks and better results on long-context benchmarks than the matched transformer with MHA at small scale.

Critical weaknesses

My main concern is that the practical benefit is not fully demonstrated.

  1. The memory savings apply between turns, but the paper does not report the time needed to reconstruct the KV cache when a conversation resumes. It would be useful to measure this overhead for different context lengths and answer lengths. This is specifically useful for long tasks such as coding.
  2. The experimental scale is small: all models have fewer than 1B parameters. The motivation is to reduce memory requirements in long-context applications, but the experiments do not show whether the approach works well for larger models, where these constraints become more severe. In particular, it is unclear whether such small per-layer extensions remain sufficient to preserve quality t larger scale. Without larger-scale experiments, I find it difficult to assess the practical relevance of the approach.
Questions for authors
  1. Could authors provide large scale experiment? This will strength the paper significantly.
2 comments

I have not received a response, so I will maintain my score.

Please see the combined response to reviewers. Specifically:

KV rematerialization cost: "The Extender materializes kv from x* at the start of each turn. The measured cost of this materialization was not provided in the paper." We did not include timing results as it varies with architecture, model size and context length. The cost is two linear projections per token in the prefix, per layer: Wk and Wv. Compared to using an existing cache in HBM, there is an added per-turn materialization cost, which is then amortized across the input and output tokens of the turn. Compared to prefill, prefill includes the materialization cost (projecting residual to K/V), but adds the remainder of attention (including softmax), and the full FFN forward. Because the ratio of prefill to rematerialization FLOPs depends on the prefix length, for 1k tokens, the ratio is 6x, at 10k tokens: 9x, 100k tokens: 36x, 1M tokens: 300x. For prefilling long prefixes, the vast majority of FLOPs are spent on attention. Rematerialization does not perform attention. The ratios above does not take memory bandwidth limits into account: attention is generally memory bound, which grows this ratio further.

Experimental scale: "900M parameters is too small, can you provide results from bigger models?" Indeed, 900M parameters is not a large model, but the results from 200M to 900M consistently demonstrated scaling. Resource constraints prevented us from training larger models by the time of submission. We have since trained a 3B extender on 40+ billion tokens, and can provide results from that in the camera ready. The model continues to scale to 3B. Again, resource constraints prevented us from reaching 60B tokens so far, and we have no matched Transformer at this size. However, validation loss 2.167 and CORE 34.84 for the 3B model line up with Chinchilla predictions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.