acceptodds
Under review as a conference paper at ICLR 2027

Depth Memory

63 views

Abstract

Residual connections propagate information over depth by additively accumulating layer outputs. Recently proposed alternatives, such as Manifold-Constrained HyperConnections (mHC) and Attention Residuals (AttnRes), instead use more expressive depth aggregation mechanisms that improve language modeling performance at scale. In this work, we consider the mixing of representations over depth a problem of storing and retrieving information across layers. Building on this perspective, we propose Depth Memory (DM), a framework that interprets the standard transformer, as well as mHC and AttnRes, as propagating information forward using different streams. One such stream can be a memory stream, which is queried to enrich the input to every layer, and whose output is subsequently written back into the memory for use by later layers. For AttnRes, this memory is a growing cache of explicit layer representations. For mHC, it is an implicit representation stored in an expanded stream. We then instantiate DM as a Linear Depth Memory (LDM), which propagates forward three streams: a standard residual stream, a memory stream, and an inverse covariance stream. We formulate the dynamics of the memory stream as a depth regression problem that is solved online via recursive least squares. We furthermore propose a hardware-aware implementation of LDM that reduces peak activation memory and memory traffic, enabling high-throughput training and inference. LDM consistently achieves the lowest perplexities among the evaluated PreLN, mHC and AttnRes baselines at the 400M and 1.5B parameter scales, alongside competitive zero-shot accuracy, showcasing the practical utility of our framework.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Transformers, Depth Mixing, Associative Memory, Residuals

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TW9ROpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Residual connections have remained the standard mechanism for information propagation across network depth since ResNet, while recent methods—including (Manifold-Constrained) HyperConnections, Attention Residuals, and Deep Delta Learning—have explored richer alternatives. Building on this line of work, the paper introduces Depth Memory, a general framework that describes depth mixing as reading from and writing to memory streams and provides a unified perspective on PreLN, mHC, and AttnRes. It then instantiates this framework as Linear Depth Memory (LDM), which maintains residual, associative-memory, and inverse-covariance streams. LDM formulates the aggregation of layer representations as a depth-regression problem, recurrently maintains its exact weighted least-squares solution using RLS, and introduces a hardware-aware implementation to reduce activation storage and memory traffic. Experiments including pretraining at the 1.5B scale compare LDM with strong PreLN, mHC, and AttnRes baselines, demonstrating competitive language modeling quality and a distinct quality–efficiency trade-off.

Critical strengths
  1. A coherent and novel synthesis of depth mixing and associative memory. The paper introduces a clean Depth Memory framework that unifies PreLN, AttnRes, and mHC through explicit read and write operations over depth. While applying sequence-mixing mechanisms along network depth has precedent in Deep Delta Learning, and recursive least squares itself is established in MesaLayer, LDM combines these ideas in a distinct way: it preserves the standard residual stream, maintains a separate fixed-size associative memory over layer representations, and recurrently maintains the exact solution to a weighted depth-regression objective. This provides a principled alternative to both explicit attention over previous layers and direct modification of the residual state.

  2. Strong architectural and systems co-design. The paper identifies the activation-storage and memory-traffic costs as major bottlenecks for making a recurrent depth memory practical and addresses them through state unwinding, windowed updates, and specialized kernels. Although the measured accuracy and throughput gains over strong depth-mixing baselines are modest, the work delivers a well-integrated framework, concrete architecture, and practical implementation. This combination establishes a credible foundation for developing more capable depth-memory mechanisms in future work.

Critical weaknesses

The empirical improvements are modest relative to the added architectural and implementation complexity. Although LDM generally achieves favorable perplexity and throughput, its accuracy gains over strong alternatives such as mHC and Block AttnRes are not consistently substantial, and the main quality evaluation is limited to 400M and 1.5B models without repeated runs or uncertainty estimates. Meanwhile, LDM introduces an additional memory state, an inverse-covariance stream, specialized training logic, and hardware-specific kernels. It therefore remains unclear whether the full RLS-based design offers a sufficiently strong practical advantage over simpler depth-mixing mechanisms.

Questions for authors

Q1. Although the appendix reports peak allocated memory across training, prefill, and decoding, the paper provides no cross-method measurement of memory traffic, despite traffic reduction being one of the main contributions. Could the authors report a component-wise memory breakdown read/written for LDM, mHC, and AttnRes in each execution phase? More generally, can the authors empirically isolate the proposed representational advantage of LDM’s explicit associative memory, rather than attributing small end-to-end differences to the memory representation alone?

Q2. The quality differences among LDM, mHC, and Block AttnRes are relatively modest, while the main 400M and 1.5B results appear to rely on a single pretraining run per configuration. Were any experiments repeated with different random seeds, or can the authors provide an estimate of run-to-run variation? Evidence that the reported improvements exceed ordinary pretraining variance would materially strengthen the empirical claim.

Q3. LDM is motivated partly by the growing representation cache of AttnRes and the limited implicit memory representation of mHC. However, Block AttnRes substantially mitigates the growing-cache problem and uses less peak memory than LDM in prefill and decoding settings, while the quality differences among LDM, Block AttnRes, and mHC remain relatively small. In which practically relevant regimes does LDM provide a clear Pareto improvement in quality, throughput, and memory? A matched-resource comparison under equal peak memory, mixer-state size, or throughput would help isolate the benefit of LDM’s explicit associative memory.

4 comments
AuthorsOpenReview ↗

Response to Reviewer TW9R (Part 1/2)

We thank the reviewer for appreciating the merits of our paper and for the interesting questions and suggestions. We address them below. Table and figure numbering refers to the revised manuscript, where changes are marked in blue.

W1: Practical gains versus complexity

Complexity We first clarify that the implementation complexity of LDM is also present in other methods. mHC changes the residual stream itself and relies on custom logic, and the same holds for AttnRes with its cache of earlier representations. Both depend on custom fused kernels and custom logic to achieve high throughput.

Regarding your question about “hardware-specific kernels”, both the mathematical formulation of LDM and our algorithmic improvements, including unwinding and windowing (Section 3.3), are independent of the GPU architecture. In our paper, we specify the hardware supported by our released kernels (Appendix F.6) because low-level implementations, which we provide for reproducibility, generally rely on hardware-specific instructions. However, as our method relies on standard primitives, it can swiftly be extended to other hardware.

Gains Aside from our main experiments, we provide new evidence that the gains are consistent. LDM achieves the lowest loss across five random seeds (400M scale, 10B tokens; Table 9 in the revision), and a lower loss than Block AttnRes at every 400M DepthBench aspect ratio (Section 4.3, new in the revision). It is also the best method in all 1.6B DepthBench runs. We report these results in detail under Q2 below.

Q1: Memory traffic and the memory representation

Thank you for this question.

Memory traffic Our traffic reduction claim concerns the optimized LDM implementation relative to a naive implementation of LDM (Section 3.3). State unwinding avoids saving every memory matrix for backward, while windowed writes reduce repeated writes of that matrix: the memory matrix is written once per window instead of once per layer, which reduces the forward accesses to LDM's state at the cost of other secondary accesses. With a correct window size (E=4E=4 in our experiments) this reduces overall traffic. Kernel fusion further reduces it.

We are happy to share the requested cross-method breakdown (Table 14 in the revision, Appendix F.4). We follow similar analytical conventions to the mHC and AttnRes papers and assume fully fused implementations. Consider the 1.5B configurations: depth blocks of eight layers for Block AttnRes, four residual streams for mHC, and Dk=8D_k=8, E=4E=4 and G=16G=16 for LDM. Then, the forward memory traffic, i.e., the values that the depth mixing step reads and writes per token and sublayer, as a multiple of the model hidden size DD (as AttnRes and mHC report traffic) is:

MethodReadWrittenTotal
PreLN2D2DDD3D3D
Block AttnRes3.9D3.9D2D2D5.9D5.9D
mHC13D13D5D5D18D18D
LDM14.8D14.8D5.3D5.3D20.1D20.1D

LDM's memory traffic is close to that of mHC despite its 2× larger state. As can be inferred in Appendix F.4, the multiplier on DD stays mostly constant across scales. Table 14 additionally breaks LDM's count into its components.

Benefit of the memory representation Our ablations already isolate the contributions of the individual components of LDM (Table 3, discussed in Section 4.2). We find that keeping the residual alongside the memory is crucial in our configurations (Section 4.2(a)). Regarding the advantages of specific memory representations, this is studied in our recurrence ablations (Section 4.2(b); done at 400M scale and 15B tokens; see Table 3 for results), where we test different memory matrix updates (our main one is based on RLS, and two simpler ones, Delta and Outer). We have repeated these across five seeds to ensure consistency. The following table shows the final validation loss of each update and seed (added as Table 10 in the revision, Appendix D.2):

Seed4243444546
LDM (RLS)2.59872.59852.60182.59862.6028
Delta2.60282.60122.60802.60862.6085
Outer2.60372.59932.61002.60112.6038

RLS outperforms Outer and Delta in every run, suggesting that what the memory represents (in this case, RLS corresponds to the exact solution to the depth regression problem we propose) is important.

Finally, our original submission already isolates the size of the memory itself. With Dk=4D_k=4, LDM's memory matrix has the same 4D4D entries per token as mHC's expanded residual state. This smaller configuration still achieves lower validation, WikiText-2, and LAMBADA perplexity than mHC, alongside higher average zero-shot accuracy (Table 2, Section 4.1).

AuthorsOpenReview ↗

Response to Reviewer TW9R (Part 2/2)

Q2: Consistency of LDM's gains across seeds and settings

We thank you for raising this concern. We are happy to provide evidence that LDM's gains persist across settings and seeds.

First, we repeated our main comparisons (Section 4.1) at 400M scale across five random seeds, on 10B tokens to fit these runs within the rebuttal period. LDM achieves the lowest loss among all competitors on every seed. The following table shows the final validation loss of each method and seed (Table 9 in the revision, Appendix D.2):

Seed4243444546
PreLN2.69512.69122.70222.69442.6921
Block AttnRes2.64182.63652.64122.65282.6489
mHC2.65312.64292.65062.64422.6404
LDM2.63032.62362.64012.62802.6262

We have also included new results on DepthBench (Section 4.3, new in the revision), where we compare LDM against strong baselines at different width/depth aspect ratios. At 1.6B, LDM achieves the lowest loss among all reported methods at every aspect ratio (Figure 7). At 400M, it also achieves the lowest loss among all reported methods at four of the five aspect ratios, and a lower loss than Block AttnRes at every one. The following table shows the final validation loss at each depth for a fixed budget of 400M parameters (plotted as perplexity in Figure 7):

# Layers (400M)1620242832
Block AttnRes2.7342.7152.7182.7262.730
LDM2.6992.6842.6902.6962.708

These experiments confirm that LDM's gains persist across model shape, size and random seeds.

Q3: Matched-resource comparison

We first clarify that we describe the growing cache of AttnRes as a property of that method within our framework rather than as the motivation for LDM.

Matched FLOPs Although we acknowledge that different baselines have different peak memory overhead, our comparisons are already approximately FLOPs-matched (Table 7, Appendix C.1): all depth mixers add small, comparable matmul costs relative to PreLN. There, LDM leads (Table 2), further supported by the new experiments we discuss in our answer to Q2.

Matched memory matrix size In Section 4.1 (concretely, Table 2), we already included a comparison where we match the memory matrix size of mHC. The Dk=4D_k=4 configuration discussed under Q1 matches the number of entries in LDM's memory matrix to mHC's expanded residual state, and it still achieves lower perplexities than mHC. It also achieves the best LAMBADA and WikiText-2 PPLs when excluding our Dk=8D_k=8 LDM configuration.

Matched training time Although we haven't conducted specific experiments where we match the training time explicitly, we are happy to provide an analysis based on our existing 1.5B runs (Section 4.1) and the throughput of each baseline. We first measure the training throughput of the 1.5B models as in our main experiments with the protocol of Appendix C (global batch size of 1M tokens with sequences of 2K tokens on 8 B200 GPUs, median of three runs):

MethodTokens/s per GPURelative to Block AttnRes
PreLN86.1K1.139
Block AttnRes75.6K1.000
mHC60.3K0.797
LDM68.7K0.909

In our 1.5B runs, LDM reaches Block AttnRes's final loss with 8.6% fewer tokens (Figure 4). Adjusting for wall-clock time, the token efficiency gain offsets the throughput overhead: LDM reaches Block AttnRes's final loss in about the same GPU-hours (0.5% more). However, this comparison disadvantages LDM, since it uses LDM's loss partway through the original learning rate schedule, designed for the full token budget, before the learning rate has completely decayed. Existing literature predicts that a time-matched comparison with the schedule adjusted to the available time budget would result in a lower final loss for LDM [1, Figure A1; 2, 3].

We sincerely hope that we have addressed the concerns of the reviewer satisfactorily in the revised version and would kindly ask them to update their score accordingly.

[1] Hoffmann et al. Training Compute-Optimal Large Language Models

[2] Hägele et al. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations

[3] Tissue et al. Scaling Law with Learning Rate Annealing

Thank you for the effort making a thorough rebuttal that helped clarifying many aspects from my concerns. I keep my positive rating on this submission.

Thanks for engaging with our rebuttal. We are glad many of your concerns are solved and thank you for the positive assessment.

Reviewer_grBcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper frames cross-layer information propagation as memory reading and writing. Linear Depth Memory (LDM) augments the residual stream with a fixed-size per-token memory and an inverse-covariance state, using recursive least squares (RLS). State unwinding and windowed writes reduce storage and memory traffic. Experiments at 400M and 1.5B show improved perplexity over PreLN, mHC, and AttnRes, with competitive zero-shot accuracy.

Critical strengths
  1. Adapting RLS to the already sequential depth axis is well motivated. The regression-based memory design, together with state unwinding and windowed writes, provides a useful architectural and systems contribution.

  2. The main comparisons use matched backbones and individually tuned learning rates. LDM improves perplexity at both scales with about 1% additional parameters over PreLN (Table 4). At 1.5B, validation perplexity is 9.15, versus 9.25 for mHC and 9.26 for Block AttnRes (Table 2).

Critical weaknesses
  1. The training-time benefit over Block AttnRes is not established. LDM has lower training throughput (Figure 7) and higher peak training memory: 94.0 versus 80.3 GiB per GPU at 1.7B with 2K sequences (Table 6). Figure 4 shows token efficiency, but quality and throughput use different model configurations, leaving the net time-to-quality benefit unclear.

  2. The added value of RLS over simpler updates needs stronger support. Its 400M cross-entropy gains over delta-rule and outer-product updates are 0.0041 and 0.0050 nats (Table 8). These 15B-token ablations use one seed and one shared peak learning rate. Without variability estimates and efficiency comparisons, whether these gains justify the covariance-state overhead remains unclear.

Overall, I lean toward weak rejection because the practical training benefit over Block AttnRes and the robust incremental value of RLS remain insufficiently established. Evidence addressing these comparisons could change my recommendation.

Questions for authors
  1. Using the existing 1.5B runs and timing measurements for that configuration, how does validation loss versus GPU-hours compare with Block AttnRes? Does the token-efficiency gain offset the throughput overhead?

  2. How robust are the RLS gains in Table 8 to peak learning rate and random seed, and what throughput/memory overhead does RLS add over the simpler updates?

4 comments
AuthorsOpenReview ↗

Response to Reviewer grBc (Part 1/2)

We thank the reviewer for appreciating the merits of our paper and for the interesting questions and suggestions. We address them below. Table and figure numbering refers to the revised manuscript, where changes are marked in blue.

W1/Q1: Quality–efficiency trade-off

Thanks for raising this point.

Throughput We apologize for the confusion about the different model sizes. As indicated in Appendix C, both models already use the same backbone: the 1.7B benchmark retains the Qwen native 151K vocabulary instead of 50K. The following table shows the training throughput of the 1.5B models as in our main experiments, measured with the protocol of Appendix C (global batch size of 1M tokens with sequences of 2K tokens on 8 B200 GPUs, median of three runs):

MethodTokens/s per GPURelative to Block AttnRes
PreLN86.1K1.139
Block AttnRes75.6K1.000
mHC60.3K0.797
LDM68.7K0.909

While your observations regarding time and memory are correct, we note that time and memory depend on hardware and implementation. Our algorithmic contributions lay the groundwork for efficient LDM implementations, but there is room for further optimization, e.g., using activation checkpointing.

Since large-scale optimization can reduce overhead and timings are not universal, model quality is typically compared at matched FLOPs instead of wall-clock time of a specific implementation. The different baselines are already approximately FLOPs-matched (Table 7, Appendix C.1): all depth mixers add small, comparable matmul costs relative to PreLN.

Time-matched comparison Even so, we are happy to provide an estimation from our existing 1.5B runs (Section 4.1) and the above timing measurements. In these runs, LDM reaches Block AttnRes's final loss with 8.6% fewer tokens (Figure 4). Adjusting for wall-clock time, the token efficiency gain offsets the throughput overhead: LDM reaches Block AttnRes's final loss in about the same GPU-hours (0.5% more). However, this comparison disadvantages LDM, since it uses LDM's loss partway through the original learning rate schedule, designed for the full token budget, before the learning rate has completely decayed. Answering your question, prior work shows that a time-matched comparison with the schedule adjusted to the available time budget would result in a lower final loss for LDM [1, Figure A1; 2, 3].

Consistency of the results We also note that the gains are not marginal and are consistent. We repeated our main 400M experiments (Section 4.1) across five seeds, but on 10B tokens in order to fit these repetitions within the rebuttal period. LDM achieves lower loss than Block AttnRes and other baselines in every run. The following table contains the final validation loss of each method and seed (Table 9 in the revision, Appendix D.2):

Seed4243444546
PreLN2.69512.69122.70222.69442.6921
Block AttnRes2.64182.63652.64122.65282.6489
mHC2.65312.64292.65062.64422.6404
LDM2.63032.62362.64012.62802.6262

Furthermore, at 1.5B, LDM achieves the lowest loss at each of the three peak learning rates in our sweep (Table 8, Appendix D.1). Additionally, we have added new experiments following DepthBench (Section 4.3, new in the revision), where we test different model width/depth aspect ratios. At 1.6B, LDM achieves the lowest loss among all reported methods at every aspect ratio (Figure 7). At 400M, it also achieves the lowest loss among all reported methods at four of the five aspect ratios, and a lower loss than Block AttnRes at every one. The following table shows the final validation loss at each depth for a fixed budget of 400M parameters (plotted as perplexity in Figure 7):

# Layers (400M)1620242832
Block AttnRes2.7342.7152.7182.7262.730
LDM2.6992.6842.6902.6962.708

In conclusion, LDM's gains are consistent across sizes, shapes, settings, and seeds. Our analysis suggests that this also holds when accounting for training time.

AuthorsOpenReview ↗

Response to Reviewer grBc (Part 2/2)

W2/Q2: Contribution of RLS

Thank you for raising this point, which we agree is important. We first want to clarify that our contributions are not exclusively the RLS formulation and are two-fold: we introduce a framework encompassing depth mixing methods as well as a depth regression objective that LDM solves online, and RLS is just an instantiation that does so exactly. Delta and Outer provide approximate solutions which we agree also work well. To assess the consistency of the RLS advantage, we repeated our recurrence ablations (Section 4.2(b), Table 3), which test different depth recurrences at 400M scale and 15B training tokens across five seeds. The following table (added as Table 10 in the revision, Appendix D.2) shows the final validation loss of each update and seed:

Seed4243444546
LDM (RLS)2.59872.59852.60182.59862.6028
Delta2.60282.60122.60802.60862.6085
Outer2.60372.59932.61002.60112.6038

RLS outperforms Outer and Delta in every run. These gains are consistent and not just marginal, though we acknowledge they do not account for most of LDM's improvement at the scales we can ablate feasibly. Motivated by sequence modeling results (Section 4.2(b)), we expect exact regression updates to matter more at greater scale or depth. The choice of RLS is further justified by its low overhead, as discussed below.

Overhead RLS adds an inverse covariance matrix per group, which is much smaller than the main memory matrix. The complexity of the Read operations in all recurrences is the same, and only the Write operations differ. The additional matrix multiplication FLOPs in these Writes are negligible. The following table (Table 15 in the revision, Appendix F.5) shows the Write FLOPs of each update per layer and token, and their share of the forward FLOPs of PreLN at 1.5B (counted as in Table 7, Appendix C.1):

Updatewrite FLOPs% of fwd at 1.5B
Outer product2DDk2DD_k0.06%
Delta rule4DDk4DD_k0.11%
RLS4DDk+4GDk24DD_k+4GD_k^20.12%

We additionally measured all three recurrent updates on a shared Triton backend for fairness. These measurements are done with a 1.5B backbone and our training settings (8 B200 GPUs, protocol of Appendix C). The following table reports the training throughput per GPU and the peak memory allocated per GPU. We find that the differences are small:

UpdateTokens/sRelativePeak memory
LDM (RLS)61.6K1.00×113.5 GiB
Delta rule63.1K1.024×109.8 GiB
Outer product63.4K1.029×109.8 GiB

Note that we use a shared Triton backend (which we used to run the ablations) also for LDM so that the speed comparison is fair (otherwise, with our additional optimizations in our main LDM implementation, LDM would actually be faster and use less memory).

We expect the overhead differences to further decrease with scale, given that the cost of a transformer grows quadratically in DD, and the additional RLS term vanishes in comparison. In conclusion, while we agree that RLS doesn't account for most of LDM's improvements, these results justify RLS as the most reasonable choice.

We sincerely hope that we have addressed the concerns of the reviewer satisfactorily in the revised version and would kindly ask them to update their score accordingly.

[1] Hoffmann et al. Training Compute-Optimal Large Language Models

[2] Hägele et al. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations

[3] Tissue et al. Scaling Law with Learning Rate Annealing

After reading the rebuttal, I believe all my concerns have been adequately addressed. Therefore, I have decided to raise my score.

We thank you for the interesting suggestions and questions, which have helped strengthen our submission. We are glad all your concerns have been addressed, and thank you further for raising your score.

Reviewer_HsGsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Depth Memory (DM), a framework that interprets information propagation across Transformer layers as a memory read/write problem. The framework provides a common perspective on standard residual connections, mHC, and Attention Residuals.

The authors further propose Linear Depth Memory (LDM), which maintains a standard residual stream, a matrix-valued associative memory, and an inverse covariance state. Memory updates are formulated as a weighted regression problem over depth and solved recursively using Recursive Least Squares (RLS). To make the method practical, the authors develop hardware-aware implementations based on state unwinding and windowed writes.

The method is evaluated through language-model pretraining at the 400M and 1.5B scales, using 50B and 100B training tokens, respectively. LDM achieves the lowest validation perplexity among the evaluated PreLN, mHC, and AttnRes baselines, while maintaining competitive zero-shot accuracy. Additional experiments investigate architectural choices and training/inference efficiency.

Critical strengths

1. A well-motivated approach to improving information propagation across depth.

The paper offers a coherent interpretation of cross-layer information aggregation through memory read/write operations. Its LDM instantiation is particularly interesting because it preserves the standard residual pathway while introducing an additional associative memory, rather than replacing residual propagation entirely. The connection to online regression provides a principled derivation of the memory update. This is a meaningful architectural direction that extends ideas from sequence-level associative memory to depth-wise information processing.

2. Substantial empirical evaluation and attention to practical implementation.

The authors conduct controlled pretraining experiments at two model scales, with matched backbone configurations and learning-rate tuning for competing methods. LDM consistently achieves the best validation perplexity among the evaluated baselines. The paper also provides extensive architectural ablations, detailed implementation specifications, and GPU throughput/memory benchmarks. The hardware-aware techniques are a valuable contribution, particularly because naive implementations of matrix-valued depth memories could otherwise incur substantial storage and memory-traffic overhead.

Critical weaknesses

1. The empirical contribution of RLS over simpler associative-memory updates is relatively small.

A central methodological contribution is the use of RLS to obtain the exact solution of the depth-regression objective. However, Table 8 shows that replacing RLS with a Delta Rule or a simple outer-product update increases the held-out loss by only 0.0041 and 0.0050 nats, respectively, in the 400M/15B-token setting. Furthermore, disabling the learned write weights or forgetting factors has a negligible effect.

These findings suggest that much of the improvement may originate from the additional associative memory and its addressing mechanism, rather than the exact RLS solver itself. Since RLS introduces additional state and computational complexity, the paper would benefit from stronger evidence that this additional complexity is justified. In particular, the corresponding comparison at the larger model scale and an assessment of run-to-run variability are missing.

2. The practical quality-efficiency trade-off is not sufficiently established against efficient baselines.

Although LDM consistently improves perplexity, the margins over Block AttnRes are relatively modest: validation perplexity improves from 12.12 to 12.08 at 400M and from 9.26 to 9.15 at 1.5B. The corresponding average zero-shot accuracy differences are also small.

Meanwhile, Figure 5 indicates that LDM has lower training throughput than Block AttnRes, and Table 6 shows higher peak memory consumption in several training and inference settings. For example, at 1.7B, peak training memory is 94.0 GiB for LDM versus 80.3 GiB for Block AttnRes, while prefill memory is 14.7 GiB versus 11.3 GiB.

Therefore, while the results support improved sample/token efficiency, it remains unclear whether LDM offers a favorable trade-off in terms of wall-clock training time, total computational cost, or deployment memory. This is important for assessing its practical significance relative to simpler and more efficient alternatives.

Questions for authors

Q1. How much of LDM's improvement can be attributed specifically to RLS?

Table 8 indicates only a small difference between RLS and simpler Delta Rule or outer-product updates. Is this difference consistent across random seeds and training scales? Could the authors provide additional evidence, potentially through a limited ablation at the larger model scale, demonstrating that the inverse covariance state and exact regression updates offer a meaningful advantage over simpler associative-memory mechanisms?

Q2. Does LDM retain its advantage under a matched computational budget?

Figure 4 demonstrates improved token efficiency, whereas Figure 5 suggests lower training throughput than Block AttnRes. Could the authors compare validation loss or perplexity as a function of GPU-hours or estimated training FLOPs, rather than training tokens alone? Even a comparison constructed from the existing training curves and measured throughput would help establish whether the quality improvements compensate for the additional runtime and memory overhead.

4 comments
AuthorsOpenReview ↗

Response to Reviewer HsGs (Part 1/2)

We thank the reviewer for appreciating the merits of our paper and for the interesting questions and suggestions. We address them below. Table and figure numbering refers to the revised manuscript, where changes are marked in blue.

W1/Q1: Contribution of RLS and gating

Thank you for this question. We first want to highlight that our contributions are two-fold: we introduce a framework encompassing depth mixing methods as well as an online depth regression objective that LDM solves. We chose RLS to maintain the exact solution. However, Delta and Outer provide reasonable approximate solutions that also work well. A 1.5B ablation is infeasible within our compute budget and rebuttal timeline. However, we have repeated our main recurrence ablations (Section 4.2(b), Table 3; 400M scale on 15B tokens) across five seeds to verify that the gains are consistent. The following table includes the final validation loss of each update and seed (added as Table 10 in the revision, Appendix D.2):

Seed4243444546
LDM (RLS)2.59872.59852.60182.59862.6028
Delta2.60282.60122.60802.60862.6085
Outer2.60372.59932.61002.60112.6038

We find that RLS outperforms Outer and Delta on every seed. These gains are consistent and not just marginal, though we acknowledge they do not account for most of LDM's improvement at the scales we can ablate feasibly. Motivated by sequence modeling results (Section 4.2(b)), we expect exact regression updates to matter more at greater scale or depth. We remark that one of our main contributions is the depth regression framework and consider alternative solvers an exciting direction for future work. The choice of RLS is further justified by its low overhead, discussed below.

Overhead RLS adds an inverse covariance matrix per group, which is much smaller than the main memory matrix. The complexity of the Read operations in all recurrences is the same, and only the Write operations differ. The additional matrix multiplication FLOPs in these Writes are negligible. The following table (Table 15 in the revision, Appendix F.5) shows the Write FLOPs of each update per layer and token, and their share of the forward FLOPs of PreLN at 1.5B (counted as in Table 7, Appendix C.1):

Updatewrite FLOPs% of fwd at 1.5B
Outer product2DDk2DD_k0.06%
Delta rule4DDk4DD_k0.11%
RLS4DDk+4GDk24DD_k+4GD_k^20.12%

We additionally measured all three recurrent updates on a shared Triton backend for fairness. These measurements are done with a 1.5B backbone and our training settings (8 B200 GPUs, protocol of Appendix C). The following table reports the training throughput per GPU and the peak memory allocated per GPU. We find that the differences are small:

UpdateTokens/sRelativePeak memory
LDM (RLS)61.6K1.00×113.5 GiB
Delta rule63.1K1.024×109.8 GiB
Outer product63.4K1.029×109.8 GiB

Note that we use a shared Triton backend (which we used to run the ablations) also for LDM so that the speed comparison is fair (otherwise, with our additional optimizations in our main LDM implementation, LDM would actually be faster and use less memory).

We expect the overhead differences to further decrease with scale, given that the cost of a transformer grows quadratically in DD, and the additional RLS term vanishes in comparison. In conclusion, while we agree that RLS doesn't account for most of LDM's improvements, these results justify RLS as the most reasonable choice.

Gating We agree learned weights and forgetting factors likewise have small effects and costs at the scales we can ablate (Table 3(d), Section 4.2(d)). We retain them because their overhead is negligible and they improve expressivity.

AuthorsOpenReview ↗

Response to Reviewer HsGs (Part 2/2)

W2/Q2: Quality–efficiency trade-off

We thank you for this comment. We clarify that training time and memory depend on hardware and implementation. Our algorithmic contributions lay the groundwork for efficient LDM implementations, with room for further optimization, e.g., using activation checkpointing or other larger-scale techniques.

FLOPs-matched comparison Our main comparisons match the backbone, but they are already approximately FLOPs-matched (Table 7, Appendix C.1). All depth mixers add small, comparable matmul costs relative to PreLN.

Time-matched comparison Even though it is more common to compare in terms of parameters or FLOPs, given that timing depends heavily on the implementation and environment, we are happy to provide the requested time-matched analysis against Block AttnRes. We first report the training throughput of the 1.5B models as in our main experiments, measured with the protocol of Appendix C (global batch size of 1M tokens with sequences of 2K tokens on 8 B200 GPUs, median of three runs):

MethodTokens/s per GPURelative to Block AttnRes
PreLN86.1K1.139
Block AttnRes75.6K1.000
mHC60.3K0.797
LDM68.7K0.909

In our 1.5B runs (Section 4.1), LDM reaches Block AttnRes's final loss with 8.6% fewer tokens (Figure 4). Adjusting for wall-clock time, the token efficiency gain offsets the throughput overhead: LDM reaches Block AttnRes's final loss in about the same GPU-hours (0.5% more). However, this comparison disadvantages LDM, since it uses LDM's loss partway through the original learning rate schedule, designed for the full token budget, before the learning rate has completely decayed. Existing literature predicts that a time-matched comparison with the schedule adjusted to the available time budget would result in a lower final loss for LDM [1, Figure A1; 2, 3].

Consistency of results We repeated our main experiments (Section 4.1) at 400M parameters, on 10B tokens to accommodate them during the rebuttal period. Across five seeds, LDM achieves lower loss than Block AttnRes in every run. The following table (Table 9 in the revision, Appendix D.2) includes the final validation loss for each baseline and seed:

Seed4243444546
PreLN2.69512.69122.70222.69442.6921
Block AttnRes2.64182.63652.64122.65282.6489
mHC2.65312.64292.65062.64422.6404
LDM2.63032.62362.64012.62802.6262

While running multiple seeds at 1.5B is infeasible within our compute budget, LDM achieves the lowest loss at each of the three peak learning rates in our sweep at that scale (Table 8, Appendix D.1). We have also added new experiments on DepthBench (Section 4.3, new in the revision), in which we compare LDM against other depth mixers across different width/depth aspect ratios. At 1.6B, LDM achieves the lowest loss among all reported methods at every aspect ratio (Figure 7). At 400M, it also achieves the lowest loss among all reported methods at four of the five aspect ratios, and a lower loss than Block AttnRes at every one. The following table shows the final validation loss at each depth for a fixed budget of 400M parameters (plotted as perplexity in Figure 7):

# Layers (400M)1620242832
Block AttnRes2.7342.7152.7182.7262.730
LDM2.6992.6842.6902.6962.708

In summary, our broad results establish that LDM's gains persist across model size, shape, and seeds. Our analysis suggests that this also holds when accounting for training time.

We sincerely hope that we have addressed the concerns of the reviewer satisfactorily in the revised version and would kindly ask them to update their score accordingly.

[1] Hoffmann et al. Training Compute-Optimal Large Language Models

[2] Hägele et al. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations

[3] Tissue et al. Scaling Law with Learning Rate Annealing

Thank you for the clarifications and additional experiments, which help address my concerns. I will maintain my original recommendation.

We thank you for the positive assessment and your insightful comments, which have helped further strengthen our submission. We are glad our clarifications and experiments helped address your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We would like to thank all three reviewers for their thorough reading of our paper and for describing Depth Memory as “a well-motivated approach to improving information propagation across depth” (HsGs) and “a coherent and novel synthesis of depth mixing and associative memory” (TW9R), and the use of RLS along depth as “well motivated” (grBc). We are also glad that the reviewers appreciated our implementation and experiments, describing the hardware-aware techniques as “a valuable contribution” (HsGs) and “a useful architectural and systems contribution” (grBc), the co-design as “a credible foundation for developing more capable depth-memory mechanisms in future work” (TW9R), and the empirical evaluation as “substantial” (HsGs), and noting that “the main comparisons use matched backbones and individually tuned learning rates” (grBc).

The constructive suggestions allowed us to improve the paper, which we hope the reviewers will find to satisfactorily address their concerns and questions. We have uploaded a revised version of the paper incorporating the reviewers' suggestions. All changes are highlighted in blue. Please note that all references to sections, figures, tables and equations refer to those in the revised version. We answer to each reviewer's specific points in the corresponding responses.

We hope that, based on the answers to their questions and the improvements made in the revised version, the reviewers will reconsider their scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.