- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes linear-time looped transformer with kv replay. A looped transformer reuses a transformer block times, trading additional computation for better performance. When the token-mixing layer is a linear attention layer, the model is a linear-time looped transformer. The paper first proposes reverse replay: at loop , the linear-attention state is updated with the kv pairs that the same block generated at loops . It then proposes first-loop replay, a variant of LT2 that updates the state only with the kv pair generated in the first loop, regardless of the current loop index. The authors claim that first-loop replay retains most of the performance gain of reverse replay while saving the FLOPs of the key and value projections in later loops.
Critical strengths
The method is simple yet effective. In the first-loop replay variant of LT2, the kv pairs generated in later loops are simply replaced by the ones generated in the first loop. This introduces no additional computation. in fact, it saves the FLOPs used to project the keys and values in later loops. These properties make the method highly promising for practical applications.
Critical weaknesses
-
The authors claim that looping can turn compute into context and it can update the memory update from rank-1 in DPLR models to rank- at most. However, it is not looping that can turn compute into context. It is historical KV replay that turns compute into context. The description in the introduction is inaccurate and misleading.
-
It seems that most of the performance gain comes from reusing the kv pair generated in the first loop. This is an interesting observation, but the paper gives no explicit explanation of why the first-loop kv pair matters so much.
-
As far as I understand, the method is simple and can therefore be adapted to other linear attention variants such as KDA. Although KDA shows no noticeable improvement over GDN in the paper, experiments on LT2-KDA with both first-loop replay and reverse replay are necessary to demonstrate the generality of the method.
Questions for authors
- Do you think the increase from rank-1 to rank- is the main source of the performance gain?
Thank you for the detailed responses, additional experiments, and clarification of the attribution of the rank-T result.
The first three explanatory experiments confirm or reconfirm that (1) the trained models rely more heavily on KV pairs from earlier loops, (2) reverse full replay produces higher effective-rank updates, and (3) stopping gradients through replayed KV reduces the gains. However, I believe the mechanisms underlying the method’s effectiveness remain incompletely understood.
The first experiment demonstrates the importance of early-loop KV under the current design, but does not explain why it is particularly useful. The third experiment shows that the additional gradient pathway contributes to performance. However, stopping this gradient omits contributions from some uses of the shared KV to the gradient of the original forward computation. Some performance degradation is therefore unsurprising, and this result provides limited explanation of why first-loop KV is especially beneficial.
Similarly, the second experiment confirms an increase in effective rank without establishing a causal relationship between higher rank and better performance, as the authors themselves acknowledge. This distinction is particularly relevant because forward full replay admits the same rank bound and may also produce higher effective-rank updates, yet performs substantially worse. Its effective rank has not been reported.
Thus, both the origin of the early-loop KV advantage and the mechanism behind reverse full replay’s additional gains over first-loop replay remain unresolved.
An incomplete mechanistic explanation does not diminish the method’s demonstrated practical usefulness. The fourth experiment strengthens the evidence that replay makes additional loops useful, while the fifth experiment substantially increases my confidence in its applicability beyond GDN.
I find the method, especially first-loop replay, strikingly simple. Given its demonstrated effectiveness and the stronger evidence for broader applicability, I am raising my recommendation to weak acceptance. A more complete and convincing theoretical analysis explaining the early-loop KV advantage and the additional benefits of reverse full replay would make me consider raising my recommendation further to clear acceptance.