- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The Extender focuses on the persistent memory footprint: memory required between turns. It separates the superposition channel h focused on next-token prediction, from the concatenation channel x dedicated to attention memory. For every size class, 32 features of attention memory per layer was enough to match the Reference Transformer on short-context tasks, and outperform it on long-context tasks.
Critical strengths
The Extender is a log-structured variant of the standard Transformer architecture. It adds a concatenation channel x: each layer emits both a residual update which is added to h, and a much smaller extension which is appended to x. While both the FFN and q see h, the attention kv projections take only x as input. For the 1664-wide, 924M model, the Extender’s persistent attention memory footprint is 104x smaller than MHA.
Critical weaknesses
- The paper relies heavily on empirical observations rather than rigorous theoretical analysis to explain why separating the residual stream h and extended embedding x improves long-context performance. There is no formal proof or bound showing that the log-structured design can preserve long-range information better than a Transformer.
- The authors observe that the Extender outperforms a Transformer on multi-key tasks while the Transformer performs better on variable tracking tasks, which indicates different inductive biases. However, the paper provides no further investigation or mechanistic interpretation for this phenomenon.
- The Extender introduces KV re-projection cost during inference. The paper only qualitatively states that the overhead is amortized, but lacks detailed latency / throughput measurement on real hardware (GPU). Readers cannot see the trade-off: how much extra compute cost must be paid to achieve the 104× memory reduction.
- The empirical evaluation is incomplete: the authors only conduct comparisons with a Transformer model, and omit benchmarking against other competing algorithms, particularly existing SOTA approaches for long-context memory reduction.
Questions for authors
The work suffers from limited theoretical grounding, narrow scaling experiments, incomplete benchmark evaluation, and insufficient measurement of inference compute-memory tradeoffs. Several interesting inductive bias phenomena are observed but left unexplained.
I have not received a response, so I will maintain my score.