- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces pretraining-time Memory Sharing for Looped Transformers. The first recursion writes full-context KV; later recursions attend to that cache and their own short local window within one softmax, yielding LPT. Applying the rule to the full-attention layers of hybrid LT2 yields H-LPT. After pretraining 150M, 450M, and 1B models on FineWeb-Edu, LPT lowers validation NLL by 0.028–0.099 nats over LT at matched size and recursion count, while reducing KV memory requirements by a factor of 2.0–4.7. Window experiments, representation analyses, gradient decomposition, and a detach intervention investigate the mechanism, and GPU measurements characterize decoding capacity and speed.
Critical strengths
- S1. A simple design with clear quality–storage benefits. Relative to LT at matched size and recursion count, LPT improves NLL while substantially reducing KV storage with under 1% additional inference FLOPs. At five recursions, LPT uses approximately 4.3–4.8 times the decoding FLOPs of parameter-matched STD, about the same as LT. Appendix F further reports H-LPT's aggregate throughput at the largest feasible batch: 1.4–1.5 times STD's at three recursions and 81–88% at five, while also exposing the latency cost at equal batch size.
- S2. Mechanistic analysis includes targeted interventions. The window experiments provide useful endpoint controls: at , prediction recursions have no local history but still read shared memory and the current token's KV; at , they retain full local history and shared access still improves quality. These controls support the contribution of shared access itself. Attention, representation, and gradient analyses further support the proposed division of roles. Retraining with the shared-KV gradient path detached hurts quality and stability, providing direct evidence that this path contributes to learning.
- S3. Several robustness checks support the main finding. Experiments cover three scales, multiple recursion depths, and related architectures. Four-seed checks at S and strengthened-normalization LT baselines in Appendix C increase confidence in the central result.
Critical weaknesses
W1. The main LPT/LT comparison changes the training objective together with the architecture, so the quality gain is not cleanly attributed.
Section 4 states that LT uses adaptive loss and all other models use final loss. The NLL gain of LPT over LT in Table 1 therefore combines memory sharing with a change of training objective. The paper's justification is that LT with final loss degrades under Ouro normalization, and Appendix C provides final-loss controls with alternative normalizations. However, the main table does not include an objective-matched LPT/LT comparison. Figure 3 reports LPT with adaptive loss, showing that this configuration trains. The paper should add LPT with adaptive loss versus LT with adaptive loss to the main table, or explicitly treat final loss as part of the method and separate the two factors in the headline claim.
W2. The quality advantage is sensitive to the LT baseline configuration, and the smallest gap is within seed noise.
The LT baseline in Table 1 uses Ouro normalization and adaptive loss. Appendix C shows that alternative normalizations make LT substantially stronger, reducing LPT's lead over the best LT to 0.020–0.026 nats at S and 0.005–0.035 nats at M. The 0.005-nat gap at M / R=2 is of the same order as the seed standard deviation measured at S, 0.005 nats. The paper reports this control honestly in Appendix C, but the abstract and main text still use the 0.028–0.099-nat range against the main LT as the headline evidence. The paper should foreground the best-baseline comparison and report seed variance or repeated runs for the smallest M-scale gaps, so that the larger main-table gaps are not read as a robust effect size.
W3. The incremental contribution over PLT is not separated from the shared-memory finding.
In Table 1, PLT also lowers NLL relative to LT at every size and recursion count, and LPT's gain over PLT at a matched KV budget is only 0.007–0.038 nats. The observation that shared memory improves quality is therefore not unique to LPT; LPT's incremental gain can only be attributed to the differences in recursion scheduling and attention organization between the two designs. Section 7 acknowledges PLT as the closest design, but the paper does not separate these two layers of contribution. It should state which conclusions reproduce an existing mechanism and which are new to LPT, and adjust the "surprising effectiveness" framing accordingly. The mechanistic analysis can support the former but does not replace a quantitative attribution of the latter.
Questions for authors
N/A
2 comments
Thank you again for your time and feedback. As the discussion deadline is approaching, we wanted to follow up on our response and the updated manuscript. We would appreciate hearing whether they address your concerns, and we would be happy to clarify any remaining points.
We thank the reviewer for their comments, which give us the opportunity to clarify a few important points.
TLDR: We use adaptive loss to keep the LT (Ouro) baseline strong, following its original recipe, while LPT performs best with final loss.
Adaptive loss supervises all recursions rather than only the last one, helping stabilize Ouro's LT as depth increases. We retain this loss in the main results to keep the baseline strong. However, as shown in Figure 4 and the first paragraph of Section 6, LPT remains stable with final loss and performs better without intermediate supervision. We therefore use final loss for LPT rather than impose an objective that reduces its performance. Furthermore, as already shown in the original submission (now Figure 4), LPT outperforms LT when both use adaptive loss.
TLDR: We keep each baseline's original normalization unchanged when adding memory sharing.
For apples-to-apples comparisons, we study two popular looped architectures (Ouro, which we refer to as LT, and LT2) with and without memory sharing. LT uses Ouro's original normalization, and we adopt exactly the same normalization for LPT. Changing the normalization only for LT would introduce another architectural difference into this comparison. Nevertheless, Appendix C also tests stronger LT normalizations. These improve LT and reduce the gap, but LPT still reaches lower perplexity than the best LT normalization in every tested setting, without any normalization tuning for LPT. We also compare LT2 with H-LPT, keeping their normalization unchanged. This provides a second apples-to-apples comparison, as both use LT2's proposed normalization choice. Memory sharing helps there as well, showing that its benefit is not limited to models using Ouro's normalization.
TLDR: We show that memory sharing improves performance in sequential looped Transformers and investigate why. We also test what explains LPT's additional gain over PLT.
PLT's ablations show that sharing memory hurts performance and that local windows largely recover it (Tables 2 and 6 of their paper). We show gains over the original models with full per-recursion caches, not just recovery of performance lost to sharing. We study this across two popular sequential looped architectures (Ouro and LT2), model sizes and recursion depths. Sharing helps both without local history and with full local caches (Figure 5), so the gain is not just an effect of short windows. We also investigate why memory sharing helps through the extensive analysis in Section 6. PLT also beating LT in our experiments is consistent with this finding. We do not claim that the benefit of memory sharing is unique to LPT. We also test why LPT performs better than PLT. We train LPT with PLT's dual attention, but keep the loops sequential. This changes perplexity by at most 0.12, while PLT is up to 0.58 worse than LPT. The larger gaps therefore come mainly from PLT's parallel loops, with a smaller contribution from the attention mechanism (Appendix G, Table 9).