- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes the context-ready transformer, which adds a correction to the current token's input embedding. The correction is computed by a FFN that takes in previous position's late hidden state and the current token's embedding, effectively breaking the bottleneck of having a single token id after sampling. During inference, it does not add extra cost as the hidden can simply be cached for the next token and discarded. During training, the authors propose to avoid BPTT by running the network K times over the sequence in parallel and using the shifted output of the previous forward (like Jacobi iteration). The experiments use OWT and Wiki and train up to 400M transformers with them. Ablations cover conversion of pretrained transformers via a zero-initialized correction FFN, and a windowed-attention pointer-chasing task under full BPTT.
Critical strengths
- The method, implementation-wise, is quite elegant and not invasive to current public SOTA transformer models, maintaining most of the optimized inference stacks like KV cache management, attention backends, etc. The authors empirically showed that the parallel training (Jacobi iteration) mostly matches the full BPTT (under 0.01 PPL) in the config sweep.
- The ablation shows that most of the gain comes from the correction step. E.g. include using residual and extra FFN under iso-params giving minimal benefits.
- The authors are open about the limitations: single runs, no iso-FLOP baselines, and prefill cost.
Critical weaknesses
- The training FLOPs are not well controlled and missing critical baselines. The D-depth model is trained with DK FLOPs essentially. If the argument is more about effective depth, then a strong loop transformer equivalent should be included for comparison. Or the argument should be iso-param mostly, but again loop transformers also fall into this category. The minimal gain is obvious as K=2 gives around 90% of improvement when we go from K=1 to K=5, which has been observed in most of the loop-flavored works.
- Most of the theoretical results have already been shown in previous works on loop transformers and similar works. E.g. Theorem 3 is essentially Banach fixed-point theorem and lemma 1 is the standard nilpotency argument for triangular Jacobi iteration. Having these theories is awesome but I wouldn’t consider these as the main contribution for the paper. Efficiency claims are made under batch size = 1, which is usually not the case for most applications.
- Given the change is minimal (which itself is a huge advantage of the method), the paper should consider scaling up to larger models (at least going beyond 1B). Moreover, since it is zero-init, it should be able to finetune on existing large checkpoints.
- There are many related works that are quite similar to this one. Full bandwidth network (a) is probably the closest one that is not directly compared (or discussed). Others include Coconut (b), Block-Recurrent Transformers (c), and TransformerFAM (d). The FBT for example uses almost the same tricks and additive injection, which is initialized from a pretrained checkpoint (so the zero-init is even more well motivated)
Questions for authors
- How does this compare to loop transformers and FBT under a relatively fair setting?
- Can you quantify how far the information carried over the last hidden somehow because I feel like this might help explain why K=2 can provide most of the gains already. I remember doing similar ideas like using the past N hiddens and an attention-like probing, which shows that the immediate last token’s hidden has most of the attention weight, suggesting that previous position’s last hidden carries most of the information needed for improvement, and can even better explain why the Jacobi-like iteration method works.
- What would happen when the batch size becomes larger? Prefill TTFT, batched throughput etc.
- Does the gain become larger or smaller as we scale up?