- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes Depth-Recurrent Memory (DRM), a recurrent memory module whose two fixed-width states per token evolve along model depth. Each state is designed for the attention pathway and the MLP pathway, respectively, injecting their readouts controlled by LSTM-inspired gate mechanism into the residual stream. With lightweight and parallel co-design, DRM can be integrated into conventional transformer architecture seamlessly and efficiently. To support the effectiveness of DRM, the paper performs broad evaluation across several benchmarks and ablation studies.
Critical strengths
This paper is well written and easy to follow. The author performs broad evaluations across several benchmark from different categories and ablation studies to support the effectiveness of DRM. The paper also introduces detailed implementation details, and makes the method easier for pratitioners to follow
Critical weaknesses
This work is not compared with the several studies for memory-augmented transformer structure. Constructing the related work section for this would be helpful to readers to understand the positioning of this paper more precisely.
The main weakness of this work is the lack of comparison with the other baselines w.r.t time complexity, and the modest improvement or comparable performance across the benchmark. It is hard to find what should be considered the main advantage of DRM compared with conventional baseline, like mHC or AttnRes.
Questions for authors
- Could you add additional related work subsection for memory-augmented transformer structure? For example, [1] introduces recurrent memory along the sequence axis, [2] introduces the memory along the model depth, but with trainable memory bank rather than recurrent updated memory. [3] adjusts the depth-wise memory with write and read mechanism, it reconstructs the next hidden state from the memory, but DRM preserves the residual path and add attention or MLP-oriented memory update into the path.
- Compared with mHC and AttnRes, what is the main benefit of DRM? (e.g., time complexity, space complexity, downstream performance).
- Could you report the training and generation wall-clock time complexity compared with several settings? (e.g., DRM vs AttnRes and mHC, parallel vs sequential formulation, with vs without short convolution buffer in autoregressive decoding setting?)
- minor question
- As DRM is designed by fixed-size state, could DRM's stability degrade as the model becomes deeper?
- Bibtex Error for Attention Residual (Team et al. 2026—> Kimi Team, 2026)
[1] Kang et al., LM2: Large Memory Models
[2] Kim et al., MIDUS: Memory-Infused Depth Up-scaling
[3] Yu et al., MeSH: memoery-as-state-highways for recursive transformers
3 comments
Response to Reviewer x2Fc [Part 2/2]
Questions
Could you add additional related work subsection for memory-augmented transformer structure? For example, [1] introduces recurrent memory along the sequence axis, [2] introduces the memory along the model depth, but with trainable memory bank rather than recurrent updated memory. [3] adjusts the depth-wise memory with write and read mechanism, it reconstructs the next hidden state from the memory, but DRM preserves the residual path and add attention or MLP-oriented memory update into the path. [1] Kang et al., LM2: Large Memory Models. [2] Kim et al., MIDUS: Memory-Infused Depth Up-scaling. [3] Yu et al., MeSH: memoery-as-state-highways for recursive transformers.
We will add the following distinctions based on the published methods:
- LM2 uses a gated memory bank accessed through cross-attention, including memory flow across decoder blocks. DRM directly propagates two token-aligned states without token-to-memory-bank cross-attention.
- MIDUS expands pretrained models using retrieval from learned key/value banks. DRM instead updates input-dependent activation states during the forward pass, augmenting the existing attention and MLP sublayers.
- MeSH synthesizes the next computational state through read/write routing over a multi-slot buffer. DRM preserves the residual stream and adds readouts from two separate recurrent states.
These comparisons primarily explore architectural design paradigms and further possibilities for DRM. The revised manuscript includes this comparison in Section 2.3 (Memory-Augmented Transformers).
Compared with mHC and AttnRes, what is the main benefit of DRM? (e.g., time complexity, space complexity, downstream performance).
DRM enhances residual information flow through independently controlled writes, retention, and readouts while preserving the backbone residual path. Its cross-depth state storage is , versus for n-stream mHC and for Block AttnRes with blocks. These describe state storage, not peak training memory; fixed-stream mHC is also depth-independent. DRM's practical advantage is the higher overall mean reported above and lower measured training overhead below, not a claim of fewer theoretical FLOPs. Section 4.3 reports the task means. Appendix B gives the measured training overheads and the cross-depth state storage comparison, including the estimates in Table 7.
Could you report the training and generation wall-clock time complexity compared with several settings? (e.g., DRM vs AttnRes and mHC, parallel vs sequential formulation, with vs without short convolution buffer in autoregressive decoding setting?)
Relative to the standard residual baseline, sequential DRM and parallel DRM without overlap optimization both incur approximately 5% end-to-end training overhead. With MLP-side computation–communication overlap, parallel DRM's overhead falls to approximately 3% in our current non-long-context setting, versus 6% for Block AttnRes and 8% for mHC. Only MLP-side overlap is used in this measurement. Separately, the formulation ablation reports losses of 1.908 for parallel DRM and 1.905 for sequential DRM, supporting the co-design trade-off.
Generation latency, including the with/without-convolution-buffer comparison, has not been measured. The short-convolution cache requires elements: it is bounded in sequence length but grows with depth and does not replace the backbone KV cache. Removing the convolution increases training loss by an absolute 0.003, but this model ablation is distinct from a cache-versus-recomputation timing comparison. Appendix B describes the training timings and the autoregressive cache size, with links to the formulation ablation in Table 5 and the convolution ablation in Appendix G.3.
As DRM is designed by fixed-size state, could DRM's stability degrade as the model becomes deeper?
We interpret this concern as potentially referring to a decline in modeling performance, since our analysis in the paper shows improved overall gradient stability with DRM. Regarding modeling performance, we believe that current model depths remain far below the capacity limits of RNN-style architectures. Recent work on linear attention and RWKV also demonstrates the ability of recurrent architectures to model long-range dependencies. See Appendix F.2 (Training Stability Analysis) for the gradient-norm comparison in the tested pretraining setting.
Bibtex Error for Attention Residual (Team et al. 2026—> Kimi Team, 2026)
Thank you. We will correct the citation to “Kimi Team, 2026” and protect the team name as a group author in BibTeX. The revised manuscript cites Attention Residuals as Kimi Team (2026), and the corrected entry is highlighted in the References.
Thank you for the detailed response.
I appreciate the additional discussion of related work and the reported training overhead comparisons, which address part of my original concerns. However, the overall performance improvements over mHC and AttnRes remain relatively modest, and the practical efficiency advantage is not yet fully established, particularly given the lack of generation latency measurements and the dependence of the best training overhead results on overlap optimizations. While these concerns prevent me from recommending acceptance, I acknowledge that the rebuttal has clarified several important points, and I will raise my score to 2 and adjust my Soundness rating to 2 as well.
Response to Reviewer x2Fc [Part 1/2]
Thank you greatly for dedicating your time and expertise to reviewing our paper. Your comments and suggestions are truly helpful to enhance our paper. Please see our response below.
Weaknesses
Thank you. Our focus is enhancing residual-stream information flow, with recurrent memory as the mechanism. We therefore prioritize standard residual, mHC, and AttnRes as experimental baselines. Due to GPU cost constraints, we have not broadly benchmarked memory-augmented Transformers. We will add a qualitative comparison with LM2, MIDUS, and MeSH in the related work, as detailed below.
Section 3.4 and Table 1 already discuss DRM's computational scaling, while Section 3.3.3 explains the overlap-enabled execution design. For performance, the unweighted mean of the 21 task scores in Table 3 is 62.84 for DRM, versus 62.44 for mHC and 62.49 for Block AttnRes: gains of approximately 0.40 and 0.35 points. DRM also leads the language/reasoning and math/code category averages. Together with the runtime measurements below, these results indicate advantages in both computational efficiency and downstream performance in our experiments. Section 4.3 now reports these task means and points to Appendix B for the wall-clock training comparison.