acceptodds
Under review as a conference paper at ICLR 2027

The Surprising Effectiveness of Shared Memory in Looped Transformers

10 views

Abstract

Looped Transformers apply the same layers several times per token, adding compute to improve quality without more parameters. Each recursion, however, writes its own key-value cache, so memory still grows with compute. Inference-time techniques can shrink this cache at a cost in quality. We pretrain looped language models to share memory: only the first recursion writes a cache, and later recursions read it while keeping a short window of their own. Surprisingly, we find that sharing memory does not cost quality and instead improves it. At 150M-1B parameters, our Looped Prediction Transformer (LPT) and its hybrid variant set a new quality–memory frontier for looped models: with five recursions, the hybrid lowers validation NLL on FineWeb-Edu by - nats relative to a same-size standard Transformer while using -% less context memory. Through an extensive analysis, we investigate why memory sharing helps. Shared and local memory develop different representations, and later recursions attend mostly to the shared memory, which also acts as a gradient highway to the first recursion.

Primary area:
generative models
Keywords:
shared memory, looped transformers

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VLNeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces pretraining-time Memory Sharing for Looped Transformers. The first recursion writes full-context KV; later recursions attend to that cache and their own short local window within one softmax, yielding LPT. Applying the rule to the full-attention layers of hybrid LT2 yields H-LPT. After pretraining 150M, 450M, and 1B models on FineWeb-Edu, LPT lowers validation NLL by 0.028–0.099 nats over LT at matched size and recursion count, while reducing KV memory requirements by a factor of 2.0–4.7. Window experiments, representation analyses, gradient decomposition, and a detach intervention investigate the mechanism, and GPU measurements characterize decoding capacity and speed.

Critical strengths
  1. S1. A simple design with clear quality–storage benefits. Relative to LT at matched size and recursion count, LPT improves NLL while substantially reducing KV storage with under 1% additional inference FLOPs. At five recursions, LPT uses approximately 4.3–4.8 times the decoding FLOPs of parameter-matched STD, about the same as LT. Appendix F further reports H-LPT's aggregate throughput at the largest feasible batch: 1.4–1.5 times STD's at three recursions and 81–88% at five, while also exposing the latency cost at equal batch size.
  2. S2. Mechanistic analysis includes targeted interventions. The window experiments provide useful endpoint controls: at ω=0\omega=0, prediction recursions have no local history but still read shared memory and the current token's KV; at ω=4096\omega=4096, they retain full local history and shared access still improves quality. These controls support the contribution of shared access itself. Attention, representation, and gradient analyses further support the proposed division of roles. Retraining with the shared-KV gradient path detached hurts quality and stability, providing direct evidence that this path contributes to learning.
  3. S3. Several robustness checks support the main finding. Experiments cover three scales, multiple recursion depths, and related architectures. Four-seed checks at S and strengthened-normalization LT baselines in Appendix C increase confidence in the central result.
Critical weaknesses

W1. The main LPT/LT comparison changes the training objective together with the architecture, so the quality gain is not cleanly attributed.

Section 4 states that LT uses adaptive loss and all other models use final loss. The NLL gain of LPT over LT in Table 1 therefore combines memory sharing with a change of training objective. The paper's justification is that LT with final loss degrades under Ouro normalization, and Appendix C provides final-loss controls with alternative normalizations. However, the main table does not include an objective-matched LPT/LT comparison. Figure 3 reports LPT with adaptive loss, showing that this configuration trains. The paper should add LPT with adaptive loss versus LT with adaptive loss to the main table, or explicitly treat final loss as part of the method and separate the two factors in the headline claim.

W2. The quality advantage is sensitive to the LT baseline configuration, and the smallest gap is within seed noise.

The LT baseline in Table 1 uses Ouro normalization and adaptive loss. Appendix C shows that alternative normalizations make LT substantially stronger, reducing LPT's lead over the best LT to 0.020–0.026 nats at S and 0.005–0.035 nats at M. The 0.005-nat gap at M / R=2 is of the same order as the seed standard deviation measured at S, 0.005 nats. The paper reports this control honestly in Appendix C, but the abstract and main text still use the 0.028–0.099-nat range against the main LT as the headline evidence. The paper should foreground the best-baseline comparison and report seed variance or repeated runs for the smallest M-scale gaps, so that the larger main-table gaps are not read as a robust effect size.

W3. The incremental contribution over PLT is not separated from the shared-memory finding.

In Table 1, PLT also lowers NLL relative to LT at every size and recursion count, and LPT's gain over PLT at a matched KV budget is only 0.007–0.038 nats. The observation that shared memory improves quality is therefore not unique to LPT; LPT's incremental gain can only be attributed to the differences in recursion scheduling and attention organization between the two designs. Section 7 acknowledges PLT as the closest design, but the paper does not separate these two layers of contribution. It should state which conclusions reproduce an existing mechanism and which are new to LPT, and adjust the "surprising effectiveness" framing accordingly. The mechanistic analysis can support the former but does not replace a quantitative attribution of the latter.

Questions for authors

N/A

2 comments

We thank the reviewer for their comments, which give us the opportunity to clarify a few important points.

Why does LT use adaptive loss while LPT uses final loss in the main results?

TLDR: We use adaptive loss to keep the LT (Ouro) baseline strong, following its original recipe, while LPT performs best with final loss.

Adaptive loss supervises all recursions rather than only the last one, helping stabilize Ouro's LT as depth increases. We retain this loss in the main results to keep the baseline strong. However, as shown in Figure 4 and the first paragraph of Section 6, LPT remains stable with final loss and performs better without intermediate supervision. We therefore use final loss for LPT rather than impose an objective that reduces its performance. Furthermore, as already shown in the original submission (now Figure 4), LPT outperforms LT when both use adaptive loss.

Why do the main comparisons use LT with Ouro's normalization rather than the other variants?

TLDR: We keep each baseline's original normalization unchanged when adding memory sharing.

For apples-to-apples comparisons, we study two popular looped architectures (Ouro, which we refer to as LT, and LT2) with and without memory sharing. LT uses Ouro's original normalization, and we adopt exactly the same normalization for LPT. Changing the normalization only for LT would introduce another architectural difference into this comparison. Nevertheless, Appendix C also tests stronger LT normalizations. These improve LT and reduce the gap, but LPT still reaches lower perplexity than the best LT normalization in every tested setting, without any normalization tuning for LPT. We also compare LT2 with H-LPT, keeping their normalization unchanged. This provides a second apples-to-apples comparison, as both use LT2's proposed normalization choice. Memory sharing helps there as well, showing that its benefit is not limited to models using Ouro's normalization.

What does your work add beyond PLT, and what explains LPT's additional gain?

TLDR: We show that memory sharing improves performance in sequential looped Transformers and investigate why. We also test what explains LPT's additional gain over PLT.

PLT's ablations show that sharing memory hurts performance and that local windows largely recover it (Tables 2 and 6 of their paper). We show gains over the original models with full per-recursion caches, not just recovery of performance lost to sharing. We study this across two popular sequential looped architectures (Ouro and LT2), model sizes and recursion depths. Sharing helps both without local history and with full local caches (Figure 5), so the gain is not just an effect of short windows. We also investigate why memory sharing helps through the extensive analysis in Section 6. PLT also beating LT in our experiments is consistent with this finding. We do not claim that the benefit of memory sharing is unique to LPT. We also test why LPT performs better than PLT. We train LPT with PLT's dual attention, but keep the loops sequential. This changes perplexity by at most 0.12, while PLT is up to 0.58 worse than LPT. The larger gaps therefore come mainly from PLT's parallel loops, with a smaller contribution from the attention mechanism (Appendix G, Table 9).

Thank you again for your time and feedback. As the discussion deadline is approaching, we wanted to follow up on our response and the updated manuscript. We would appreciate hearing whether they address your concerns, and we would be happy to clarify any remaining points.

Reviewer_rts5OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies memory sharing in looped Transformers. In the proposed Looped Prediction Transformer (LPT), only the first recursion writes a full-context KV cache, and later recursions read this shared cache together (+ a short window for each new loop). Hence, the memory stays close to a standard Transformer. The authors pretrain models from 150M to 1B parameters on FineWeb-Edu and show that sharing does not hurt but improves quality over a full-cache looped Transformer, also in a hybrid version (H-LPT). They then analyze why, showing that later recursions mostly attend to the shared memory and that the shared cache gives the first recursion a direct gradient path.

Critical strengths

(1) I enjoyed reading this paper a lot. I really like the simple solution to the KV cache problem of looped Transformers, which I think is the most important issue right now for making them practical. Only the first recursion writes a full cache, and this does not cost quality but improves it. For example, at L size with five recursions, LPT lowers NLL over LT with 4.7× less memory!

(2) I also like the analysis a lot. The paper shows that later recursions put more and more attention on the shared memory, and that shared and local keys form different clusters. I also love the connection to the gradient flow, where reading the shared memory gives a direct path back to the first recursion, and Table 2 shows that blocking this path hurts training. I think this is brilliant and seals the contributions really well.

Critical weaknesses

(1) The paper is motivated by memory at long contexts, but all models use 4,096-token contexts and are tested on short zero-shot tasks. Since only the first recursion keeps a full cache, I wonder if tasks that need long-range retrieval suffer. I would like to see at least one retrieval-heavy test, such as passkey retrieval or RULER.

(2) All comparisons are parameter-matched, while at five recursions LPT costs 4.3 to 4.8× the decoding FLOPs of STD. It is therefore not clear how LPT compares with simply using a larger standard model at the same compute. I would like to see a FLOP-matched standard Transformer, at least at S size.

Questions for authors

(1) The ω = 0 setting looks close to the KV sharing variant of MoR (Bae et al., 2025), which the paper says costs a bit of quality, while here sharing improves over LT. What do you think explains this difference?

(2) PLT differs from LPT in two ways, parallel recursions and gated dual attention instead of one softmax. I wonder which of the two explains why LPT gets lower NLL.

2 comments

We thank the reviewer for appreciating our method and analysis and for the interesting questions.

Can you show long-context evaluations? Reducing memory might hurt there.

TLDR: Memory-sharing helps even with long context.

We test the standard Transformer, LT, LPT, LT2, H-LPT on FinePDF long documents and on RULER at our models’ full training context of 4,096 tokens. We show the results in Figure 3 in the revised manuscript. Memory-sharing does not hurt and its gains also transfer to the long-context regime.

Why does memory sharing hurt in MoR and help here?

TLDR: Results in MoR about memory sharing are nuanced and mixed. We explained this better in our new revision and below.

Results in MoR for memory sharing are on a single low-compute budget model size (360M parameters, 10B tokens), and conclusions differ across configurations of MoR. In the closest configuration to ours, they observe similar but mixed results on the only two models they try (Recursive, 98M and 157M, Cyc, in Table 15 in Appendix H.2 of MoR paper). Overall, these mixed results further motivate the need for our more comprehensive analysis on the topic. We train models at significantly higher compute budgets (up to 66x MoR's training FLOPs), based on two popular looped architectures (Ouro and LT2), and observe that memory sharing consistently helps. Interestingly, the smallest gain of LPT over LT is at w=0 at S size with R=2 (Figure 5), our setting closest to MoR's compute budget (about 1.5x its FLOPs). This suggests that training budget may be one reason for the different observations.

We have updated our discussion of MoR to better explain this difference.

Which part of PLT hurts the most? Is it the different memory-sharing (dual attention) or the parallel loops?

TLDR: The parallel loops.

We test LPT with the dual attention memory-sharing mechanism used by PLT. At this point, the only difference between PLT and LPT is that PLT runs the loops in parallel through its shifting mechanism. We find that the dual-attention mechanism hurts only slightly: LPT with PLT's gate stays within 0.12 perplexity of LPT, while PLT trails by up to 0.58 (Appendix G of our revised manuscript, Table 9). Thus, most of the gap comes from the parallel loops.

How do LPT and H-LPT compare to simply using a larger Transformer?

Using our existing training runs, we added comparisons with larger standard Transformers at matched training compute (Figure 20, Appendix H). The larger Standard reaches lower perplexity than LPT, but needs 1.8-2.6x as much context memory. H-LPT at M with two and three recursions outperforms Standard at L with about a tenth of its context memory.

More generally, looping is not primarily a way to save FLOPs, but a way to trade more compute for less memory: looped models can spend more FLOPs per token to compete with larger, more memory-hungry models. This is an attractive tradeoff on modern GPUs, where decoding is often more constrained by memory than compute. Our contribution improves this tradeoff: LPT and H-LPT improve quality and reduce context memory at roughly the same FLOPs as LT and LT2, respectively.

Thanks for the detailed response and for taking the time to address my questions. I'm keeping my positive score.

Reviewer_DFLoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper makes the observation that although in Looped Transformers training models adds more compute without increasing the number of parameters, the context memory, KV cache, still grows with compute. The authors propose a “shared memory” mechanism -- only the first recursion writes the cache and later recursions read it in addition to a short window of KV cache of their own. Pretraining the whole model with shared memory, the authors show that on 150M, 450M and 1B size models, their proposed method not only saves the context memory but also results in lower NLL in the heldout FineWeb-Edu corpus, and higher zero-shot accuracy in several downstream tasks, compared to Loop Transformers baselines.

Critical strengths
  • The paper highlights the potential improvement in saving context memory and includes this design not only at inference time, but also in training LLM as well.
  • The proposed method is simple and seems effective in the experimental study conducted in this work.
  • The presentation is generally clear and easy to follow.
Critical weaknesses
  • The sizes of the models being trained are still relatively small. In addition, the 150M (S), 450M (M), 1B (L) scaling ladder does not always show that the gain remains the same or increases as the model size increases, depending on which baseline it is compared to. The token budgets also differ, with the tokens per parameter ratio being about 132 (S), 44 (M) and 50 (L). S is over-trained compared to M and L, making the comparison difficult.
  • The analyses use S and M only (Section 8). With small-size models, it’s hard to know whether the observations remain the same for larger, more practically used models.
  • The downstream tasks in the selected benchmarks are all short-context, which is the limitation admitted by the authors. However, one could argue that restricting later recursions to a short window of their own cache (ω = 64), with only the first recursion's cache covering the full context, could make the model weaker in dealing with long-context tasks, which is crucial in agentic applications of LLMs.
Questions for authors
  • How is the token budget chosen for different sizes of the models (20B tokens for S and M, and 50B for L)?
  • Do the performance gaps (e.g., vs. LT and STD) hold at a matched tokens-per-parameter ratio or at matched training compute?
  • Why does first-recursion sharing help here when MoR (Bae et al., 2025) reported a slight cost (Sec. 7)?
2 comments

We thank the reviewer for their review and for the opportunity to clarify a few important points.

The analysis is only available for S and M sizes.

TLDR: We have now extended the main analyses to L, with the same trends and conclusions.

We initially limited these ablations to S and M because they require training new models, which is particularly expensive for looped models at L size. In the revision, we trained L models for 20B tokens (same as all S and M models) for the final vs adaptive loss, window-size, and shared-memory gradient-detachment ablations (~12k additional GPU-hours). We also extended the attention-state, PCA, and gradient-highway analyses to L size with the final 50B tokens checkpoints and updated the corresponding tables and figures. The trends are the same and our conclusions hold.

Can you show long-context evaluations? Reducing memory might hurt there.

TLDR: Memory-sharing helps even with long context.

We test the standard Transformer, LT, LPT, LT2, H-LPT on FinePDF long documents and on RULER at our models’ full training context of 4,096 tokens. We show the results in Figure 3 in the revised manuscript. Memory-sharing does not hurt and its gains also transfer to the long-context regime.

How is the token budget chosen for different sizes of the models (20B tokens for S and M, and 50B for L)?

We chose a single number of tokens that would be more than Chinchilla-optimal for S and M, and then extended to 50B for L so that this size would not be undertrained as well.

Do the performance gaps (e.g., vs. LT and STD) hold at a matched tokens-per-parameter ratio or at matched training compute?

TLDR: Yes, the gains hold at matched tokens per parameter and at matched FLOPs against LT and LT2. Against larger standard Transformers, the results show a quality-memory tradeoff.

We observe stable trends across sizes at our maximum tokens/params ratio available across sizes (Figure 19 in Appendix H of our revised manuscript). FLOPs-wise, LPT and H-LPT use roughly the same (<0.5% difference) as LT and LT2 respectively, and are better in FLOPs comparison as well. With respect to Standard, as we mentioned in our Discussion section, looping does not always benefit training FLOPs. This is expected (as costs of looping are much higher). Still, at matched training compute, H-LPT at M with two and three recursions outperforms Standard at L with about a tenth of its context memory (Figure 20, Appendix H).

More generally, looping is not primarily a way to save FLOPs, but a way to trade more compute for less memory: looped models can spend more FLOPs per token to compete with larger, more memory-hungry models. This is an attractive tradeoff on modern GPUs, where decoding is often more constrained by memory than compute. Our contribution improves this tradeoff: LPT and H-LPT improve quality and reduce context memory at roughly the same FLOPs as LT and LT2, respectively.

Why does memory sharing hurt in MoR and help here?

TLDR: Results in MoR about memory sharing are nuanced and mixed. We explained this better in our new revision and below.

Results in MoR for memory sharing are on a single low-compute budget model size (360M parameters, 10B tokens), and conclusions differ across configurations of MoR. In the closest configuration to ours, they observe similar but mixed results on the only two models they try (Recursive, 98M and 157M, Cyc, in Table 15 in Appendix H.2 of MoR paper). Overall, these mixed results further motivate the need for our more comprehensive analysis on the topic. We train models at significantly higher compute budgets (up to 66x MoR's training FLOPs), based on two popular looped architectures (Ouro and LT2), and observe that memory sharing consistently helps. Interestingly, the smallest gain of LPT over LT is at w=0 at S size with R=2 (Figure 5), our setting closest to MoR's compute budget (about 1.5x its FLOPs). This suggests that training budget may be one reason for the different observations.

We have updated our discussion of MoR to better explain this difference.

Thank you again for your time and feedback. As the discussion deadline is approaching, we wanted to follow up on our response and the updated manuscript. We would appreciate hearing whether they address your concerns, and we would be happy to clarify any remaining points.

Comments on the paper

1 comment

We thank the reviewers for their hard work and insightful questions, which strengthened our evidence. We have updated the manuscript with the new results requested by the reviewers (~17k additional H100 GPU-hours) and summarize the changes below:

  • Memory sharing also helps with long context tasks (DFLo, rts5). We test our trained models on long-document perplexity (FinePDF) and long-context tasks (RULER), using our full training context of 4096 tokens. This analysis is now at the end of the results section.
  • All analyses now include L-sized models (DFLo). We trained L models for all ablations for 20B tokens (same as all S and M models): final vs adaptive loss, window size, and shared-memory gradient detach (~12k GPU-hours). We also extended the attention-state, PCA, and gradient-highway analyses to L size with the final 50B tokens checkpoints. All tables and figures in the main paper are updated. The trends are the same and our conclusions hold.
  • PLT's gap comes mostly from its parallel loops (rts5, VLNe). To separate the effect of PLT's parallel loops from its memory-sharing mechanism, we trained LPT with PLT's dual-gate attention sharing (~5k GPU-hours). Dual-gate attention is generally slightly worse than ours, but most of the gap is due to the parallel loops. We report this in the appendix.
  • Matched-budget comparisons (DFLo, rts5). We added comparisons at matched tokens per parameter across sizes and matched training compute, including comparisons against larger standard Transformers (Appendix H).
  • Perplexity instead of NLL. We switched to PPL as it is easier to interpret.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.