acceptodds
Under review as a conference paper at ICLR 2027

Enhancing Deep Residual LLMs with Depth-Recurrent Memory

Abstract

Residual connections have been a foundational design principle in large language models (LLMs), enabling the residual stream to accumulate layer-wise information across depth through unit-weight additions. Although this accumulation provides a stable information pathway, its fixed additive form offers limited explicit and input-dependent control over the evolving residual content. Recent studies have introduced constrained mixing or attention mechanisms to make residual information flow more adaptive. However, they still rely on the residual stream or stored intermediate layer representations as the carrier of cross-layer information, without an independent layer-wise memory and an explicit mechanism for controlling what to write, retain, and expose across depth. To fill this gap, we propose Depth-Recurrent Memory (DRM), a recurrent memory module whose state evolves along model depth. DRM maintains two fixed-width memory states per token, one for the attention pathway and one for the MLP pathway, and injects their gated readouts into the residual stream as additive updates. This provides explicit recurrent memory management while preserving the standard backbone sublayers and residual pathways. We instantiate the recurrent transitions with LSTM-style gating and co-design lightweight input projections with a parallel-residual implementation to control parameter and computational overhead and enable concurrent execution with backbone sublayers. Experimental results show that DRM outperforms the standard residual baseline on 16 of 21 benchmarks, with the strongest gains in math and code. In addition, the ablation studies validate the effectiveness of core components of DRM.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Pre-Training, Residual Connections, Recurrent Memory

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_x2FcOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes Depth-Recurrent Memory (DRM), a recurrent memory module whose two fixed-width states per token evolve along model depth. Each state is designed for the attention pathway and the MLP pathway, respectively, injecting their readouts controlled by LSTM-inspired gate mechanism into the residual stream. With lightweight and parallel co-design, DRM can be integrated into conventional transformer architecture seamlessly and efficiently. To support the effectiveness of DRM, the paper performs broad evaluation across several benchmarks and ablation studies.

Critical strengths

This paper is well written and easy to follow. The author performs broad evaluations across several benchmark from different categories and ablation studies to support the effectiveness of DRM. The paper also introduces detailed implementation details, and makes the method easier for pratitioners to follow

Critical weaknesses

This work is not compared with the several studies for memory-augmented transformer structure. Constructing the related work section for this would be helpful to readers to understand the positioning of this paper more precisely.

The main weakness of this work is the lack of comparison with the other baselines w.r.t time complexity, and the modest improvement or comparable performance across the benchmark. It is hard to find what should be considered the main advantage of DRM compared with conventional baseline, like mHC or AttnRes.

Questions for authors
  1. Could you add additional related work subsection for memory-augmented transformer structure? For example, [1] introduces recurrent memory along the sequence axis, [2] introduces the memory along the model depth, but with trainable memory bank rather than recurrent updated memory. [3] adjusts the depth-wise memory with write and read mechanism, it reconstructs the next hidden state from the memory, but DRM preserves the residual path and add attention or MLP-oriented memory update into the path.
  2. Compared with mHC and AttnRes, what is the main benefit of DRM? (e.g., time complexity, space complexity, downstream performance).
  3. Could you report the training and generation wall-clock time complexity compared with several settings? (e.g., DRM vs AttnRes and mHC, parallel vs sequential formulation, with vs without short convolution buffer in autoregressive decoding setting?)
  4. minor question
    1. As DRM is designed by fixed-size state, could DRM's stability degrade as the model becomes deeper?
    2. Bibtex Error for Attention Residual (Team et al. 2026—> Kimi Team, 2026)

[1] Kang et al., LM2: Large Memory Models

[2] Kim et al., MIDUS: Memory-Infused Depth Up-scaling

[3] Yu et al., MeSH: memoery-as-state-highways for recursive transformers

3 comments
AuthorsOpenReview ↗

Response to Reviewer x2Fc [Part 1/2]

Thank you greatly for dedicating your time and expertise to reviewing our paper. Your comments and suggestions are truly helpful to enhance our paper. Please see our response below.

Weaknesses

This work is not compared with the several studies for memory-augmented transformer structure. Constructing the related work section for this would be helpful to readers to understand the positioning of this paper more precisely.

Thank you. Our focus is enhancing residual-stream information flow, with recurrent memory as the mechanism. We therefore prioritize standard residual, mHC, and AttnRes as experimental baselines. Due to GPU cost constraints, we have not broadly benchmarked memory-augmented Transformers. We will add a qualitative comparison with LM2, MIDUS, and MeSH in the related work, as detailed below.

The main weakness of this work is the lack of comparison with the other baselines w.r.t time complexity, and the modest improvement or comparable performance across the benchmark. It is hard to find what should be considered the main advantage of DRM compared with conventional baseline, like mHC or AttnRes.

Section 3.4 and Table 1 already discuss DRM's computational scaling, while Section 3.3.3 explains the overlap-enabled execution design. For performance, the unweighted mean of the 21 task scores in Table 3 is 62.84 for DRM, versus 62.44 for mHC and 62.49 for Block AttnRes: gains of approximately 0.40 and 0.35 points. DRM also leads the language/reasoning and math/code category averages. Together with the runtime measurements below, these results indicate advantages in both computational efficiency and downstream performance in our experiments. Section 4.3 now reports these task means and points to Appendix B for the wall-clock training comparison.

AuthorsOpenReview ↗

Response to Reviewer x2Fc [Part 2/2]

Questions

Could you add additional related work subsection for memory-augmented transformer structure? For example, [1] introduces recurrent memory along the sequence axis, [2] introduces the memory along the model depth, but with trainable memory bank rather than recurrent updated memory. [3] adjusts the depth-wise memory with write and read mechanism, it reconstructs the next hidden state from the memory, but DRM preserves the residual path and add attention or MLP-oriented memory update into the path. [1] Kang et al., LM2: Large Memory Models. [2] Kim et al., MIDUS: Memory-Infused Depth Up-scaling. [3] Yu et al., MeSH: memoery-as-state-highways for recursive transformers.

We will add the following distinctions based on the published methods:

  • LM2 uses a gated memory bank accessed through cross-attention, including memory flow across decoder blocks. DRM directly propagates two token-aligned states without token-to-memory-bank cross-attention.
  • MIDUS expands pretrained models using retrieval from learned key/value banks. DRM instead updates input-dependent activation states during the forward pass, augmenting the existing attention and MLP sublayers.
  • MeSH synthesizes the next computational state through read/write routing over a multi-slot buffer. DRM preserves the residual stream and adds readouts from two separate recurrent states.

These comparisons primarily explore architectural design paradigms and further possibilities for DRM. The revised manuscript includes this comparison in Section 2.3 (Memory-Augmented Transformers).

Compared with mHC and AttnRes, what is the main benefit of DRM? (e.g., time complexity, space complexity, downstream performance).

DRM enhances residual information flow through independently controlled writes, retention, and readouts while preserving the backbone residual path. Its cross-depth state storage is O(BT(d+2dm))O(BT(d + 2d_m)), versus O(nBTd)O(nBTd) for n-stream mHC and O(NbBTd)O(N_bBTd) for Block AttnRes with NbN_b blocks. These describe state storage, not peak training memory; fixed-stream mHC is also depth-independent. DRM's practical advantage is the higher overall mean reported above and lower measured training overhead below, not a claim of fewer theoretical FLOPs. Section 4.3 reports the task means. Appendix B gives the measured training overheads and the cross-depth state storage comparison, including the estimates in Table 7.

Could you report the training and generation wall-clock time complexity compared with several settings? (e.g., DRM vs AttnRes and mHC, parallel vs sequential formulation, with vs without short convolution buffer in autoregressive decoding setting?)

Relative to the standard residual baseline, sequential DRM and parallel DRM without overlap optimization both incur approximately 5% end-to-end training overhead. With MLP-side computation–communication overlap, parallel DRM's overhead falls to approximately 3% in our current non-long-context setting, versus 6% for Block AttnRes and 8% for mHC. Only MLP-side overlap is used in this measurement. Separately, the formulation ablation reports losses of 1.908 for parallel DRM and 1.905 for sequential DRM, supporting the co-design trade-off.

Generation latency, including the with/without-convolution-buffer comparison, has not been measured. The short-convolution cache requires 2LB(K−1)dm2LB(K−1)d_m elements: it is bounded in sequence length but grows with depth and does not replace the backbone KV cache. Removing the convolution increases training loss by an absolute 0.003, but this model ablation is distinct from a cache-versus-recomputation timing comparison. Appendix B describes the training timings and the autoregressive cache size, with links to the formulation ablation in Table 5 and the convolution ablation in Appendix G.3.

As DRM is designed by fixed-size state, could DRM's stability degrade as the model becomes deeper?

We interpret this concern as potentially referring to a decline in modeling performance, since our analysis in the paper shows improved overall gradient stability with DRM. Regarding modeling performance, we believe that current model depths remain far below the capacity limits of RNN-style architectures. Recent work on linear attention and RWKV also demonstrates the ability of recurrent architectures to model long-range dependencies. See Appendix F.2 (Training Stability Analysis) for the gradient-norm comparison in the tested pretraining setting.

Bibtex Error for Attention Residual (Team et al. 2026—> Kimi Team, 2026)

Thank you. We will correct the citation to “Kimi Team, 2026” and protect the team name as a group author in BibTeX. The revised manuscript cites Attention Residuals as Kimi Team (2026), and the corrected entry is highlighted in the References.

Thank you for the detailed response.

I appreciate the additional discussion of related work and the reported training overhead comparisons, which address part of my original concerns. However, the overall performance improvements over mHC and AttnRes remain relatively modest, and the practical efficiency advantage is not yet fully established, particularly given the lack of generation latency measurements and the dependence of the best training overhead results on overlap optimizations. While these concerns prevent me from recommending acceptance, I acknowledge that the rebuttal has clarified several important points, and I will raise my score to 2 and adjust my Soundness rating to 2 as well.

Reviewer_mbc1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Depth-Recurrent Memory (DRM), a recurrent memory module that evolves across model depth. DRM maintains two independent vector states for each attention and MLP layer per token in the residual stream. Across depth the memory states are modulated by input, forget, output gates and a write projection inspired by LSTMs. To reduce the amount of additional parameters stemming from the gate projections, they are chosen to be sum of a head-wise block-diagonal and a low rank projection. The residual stream after each attention and MLP layer is updated from their respective memory state with a dense output projection and a scalar multiplier. The effectiveness of DRM is demonstreated in scaling law experiments and a comparison to a standard residual, mHC and AttentionResidual baseline. DRM improves over standard residuals on 16 out of 21 downstream evaluations with strongest gains on math and code tasks.

Critical strengths
  • To the best of my knowledge the method is novel, and well motivated. It transfers the LSTM gating mechanism over time to the depth dimension.
  • The architecture introduces explicit write, retention, and readout control through separate memory pathways while preserving the backbone residual additions. The layer-specific memory transitions and their integration with the backbone are explained clearly.
  • The head-wise and low-rank projection design has a clear motivation, and the comparison with dense input projections supports the chosen parameter reduction at the tested scale.
  • The experimental coverage includes small-model scaling experiments, a substantial MoE pretraining comparison with alternative residual architectures, and several component ablations. Appendix E also includes a useful control for increased total parameter count.
  • Well written and structured. The sections describing the memory updates and the headwise and projections are easy to follow.
  • Strong ablation section, with major design decisions ablated.
  • The main comparison of model with 1.3B active and 7.9B total parameters trained for 1T tokens is large scale.
Critical weaknesses
  • The complexity and number of parameters are very well explained. The picture could be completed by further elaborating on the wall-clock runtime and activation memory. Moreover, ther parallel execution strategy is reasonable, but the paper does not state whether overlapping DRM computation with communcation was explored in these experiments.
  • The number of evaluation tasks is large, but there are no hold out validation NLL numbers provided. Given these are pretraining experiments they could provide additional signal, because downstream evals can be noisy.
Questions for authors
  • The motivation for the causal conv 1d layer to introduce a short-range local context into the write signal seems plausible. Did you also ablate this design decision?
  • The batch size of 8192 tokens seems very small for a model with 1.3B active and 7.9B total parameters.Does batch size denote sequences or tokens, is it global or per device, and what is the effective global token batch after gradient accumulation?
  • DRM adds additional computation, but in the scaling law curves (Fig. 2) the FLOPs per tier seem identical to the baseline (same x-position). How did you count the FLOPs for this experiment?
  • DRM adds a dense output projection that projects the memory state into the residual dimension. In addition, before adding the result projected vector to the residual stream, these residual stream updates from DRM are multiplied by learnable scalars. Are these essential? They allow the model even to ignore DRM completely. Did you investigate their values after training, to probe the magnitude of the residual DRM updates? I would consider this as additional analysis to appendix F.1, that investigates the memory update strength or ratio within DRM per layer.
  • Minor comment: Following your convention that lowercase bold symbols denote rows, should the gates, e.g. R_{l,t} or I_{l,t} in eq. (4) also be lower case?
4 comments
AuthorsOpenReview ↗

Response to Reviewer mbc1 [Part 1/2]

We sincerely thank you for the time and effort you have invested in reviewing our paper. Your comments and questions are very insightful and highly valuable for us to enhance the paper. Your comments are addressed point-by-point in the following.

Weaknesses

The complexity and number of parameters are very well explained. The picture could be completed by further elaborating on the wall-clock runtime and activation memory. Moreover, ther parallel execution strategy is reasonable, but the paper does not state whether overlapping DRM computation with communcation was explored in these experiments.

In our 7.9B pretraining runs, wall-clock time was approximately 5% higher for DRM without communication-overlap optimization, 8% higher for mHC, and 6% higher for Block AttnRes relative to the standard residual baseline. After applying a dedicated schedule that overlaps MLP-side DRM computation with communication, DRM's wall-clock overhead was reduced to approximately 3% relative to the same baseline. For one 8,192-token sequence, the estimated peak forward activation storage for the cross-depth state is 72 MiB for DRM, which carries two fixed-width states alongside the residual stream, compared with 24 MiB for the baseline, 96 MiB for mHC, and 312 MiB for Block AttnRes under BF16 storage. We added the timings, activation-memory estimates, and overlap setting to Appendix B under "Training Runtime and Activation Memory."

The number of evaluation tasks is large, but there are no hold out validation NLL numbers provided. Given these are pretraining experiments they could provide additional signal, because downstream evals can be noisy.

We agree that held-out NLL would provide complementary evidence. Our scaling experiments report training loss, which improves at all five tested scales but does not replace held-out evaluation. We do not report additional held-out NLL results in this rebuttal and will clarify this distinction. In the revised manuscript, Section 4.2 and the caption of Figure 2 identify the reported metric as final training loss.

AuthorsOpenReview ↗

Response to Reviewer mbc1 [Part 2/2]

Questions

The motivation for the causal conv 1d layer to introduce a short-range local context into the write signal seems plausible. Did you also ablate this design decision?

Yes. Removing the short causal convolution increases final training loss by an absolute 0.003. This supports its local-context benefit, while the depth-recurrent mechanism remains intact without it. We will add this ablation result. Appendix G.3 (Short Causal Convolution) reports the ablation using the 47M-activated-parameter scaling-law configuration.

The batch size of 8192 tokens seems very small for a model with 1.3B active and 7.9B total parameters.Does batch size denote sequences or tokens, is it global or per device, and what is the effective global token batch after gradient accumulation?

Each training sequence contains 8,192 tokens. After gradient accumulation, each optimizer step processes a global batch of 8,192 sequences across all devices, for a total of 67,108,864 tokens. We clarified the sequence length and the global batch size per optimizer step in Table 6 of Appendix B.

DRM adds additional computation, but in the scaling law curves (Fig. 2) the FLOPs per tier seem identical to the baseline (same x-position). How did you count the FLOPs for this experiment?

At each model scale, we use the standard-residual baseline’s training FLOPs for both runs in Fig. 2. This gives the two runs the same x-coordinate, while the DRM memory branches add to the actual training cost of the DRM run. We clarified the FLOP accounting in Section 4.2 (Scaling Laws) and in the caption of Fig. 2.

DRM adds a dense output projection that projects the memory state into the residual dimension. In addition, before adding the result projected vector to the residual stream, these residual stream updates from DRM are multiplied by learnable scalars. Are these essential? They allow the model even to ignore DRM completely. Did you investigate their values after training, to probe the magnitude of the residual DRM updates? I would consider this as additional analysis to appendix F.1, that investigates the memory update strength or ratio within DRM per layer.

Although these scaling factors are not structurally required, they play an important role in regulating information flow. Their statistics in the final 7.9B model are as follows:

Memory branchMinimum scaleMaximum scaleMean across layers
Attention-side0.36133.70312.3564
MLP-side1.41412.57812.1149

None of the scales is close to zero, demonstrating the necessity of the additional information injected into the residual stream. MLP scales are smaller on average, while attention scales show a stronger depth trend: the mean rises from 1.28 in layers 0–7 to 2.96 in layers 16–23, versus 1.86 to 2.19 for MLP. Thus, later layers generally assign larger scalar weights to memory readouts, especially on the attention side. We will supplement the analysis in Appendix F.1 with these results. Figure 4 in Appendix F.1 now shows the learned coefficients for all 24 layers, and Appendix G.1 reports the fixed-unit-scale ablation.

Minor comment: Following your convention that lowercase bold symbols denote rows, should the gates, e.g. R_{l,t} or I_{l,t} in eq. (4) also be lower case?

Yes. We will use lowercase bold symbols for token-level gate and decay vectors, including rℓ,tr_{\ell,t} and iℓ,ti_{\ell,t}, and uppercase bold symbols for full matrices. The definitions and equations in Section 3.2 now follow this convention, and Appendix F.1 identifies the corresponding matrix rows.

Thank you for your responses and your updates to the paper. Some of my concerns have been addressed others remain open.

I will maintain my score, but want to remark that some of my concerns remain to some extent:

  • After your clarification on the batch size, I remain worried that this now very large batch size of 67M tokens is not sufficiently justified (At 1T tokens that means about only 14,900 optimizer steps). Standard batch sizes for 8B dense models are in the range of 2M-8M (depending on training length). While DeepSeekMoE (https://arxiv.org/html/2401.06066v1/) 16B uses approximately 18.9M tokens per batch yours is still considerably larger. Did you see benefits of this large batch size with your method?

  • The scaling laws do not account for the additional FLOPs of DRM. It might be more fair to add them, which would quantify the horizontal shift in your scaling curves. It might very well be that DRM still is better than the baseline since the number of FLOPs for DRM should be relatively small, but the authors did not show this.

  • Validation NLL is still not shown. All figures were relabelled as training loss.

We apologize that our response did not fully resolve your concerns. We did make some mistakes that were previously overlooked:

  1. The official training setup uses a sequence length of 8192 and a batch size of 2048, yielding a training batch data of 16M tokens. This is the optimal batch size range derived from our full scaling law experiments, and the learning rate was set accordingly based on the same experiments. As shown in Figure 4, the final training run lasted for approximately 60k steps, which is consistent with the 16M-token batch-size setting. Nevertheless, the batch size listed in Table 6 is incorrect, and we will revise it.
  2. We will revise the scaling law plots, even if the shift along the horizontal axis is not very pronounced.
  3. We will clearly label all loss curves as training curves rather than validation NLL curves.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.