- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Residual connections have remained the standard mechanism for information propagation across network depth since ResNet, while recent methods—including (Manifold-Constrained) HyperConnections, Attention Residuals, and Deep Delta Learning—have explored richer alternatives. Building on this line of work, the paper introduces Depth Memory, a general framework that describes depth mixing as reading from and writing to memory streams and provides a unified perspective on PreLN, mHC, and AttnRes. It then instantiates this framework as Linear Depth Memory (LDM), which maintains residual, associative-memory, and inverse-covariance streams. LDM formulates the aggregation of layer representations as a depth-regression problem, recurrently maintains its exact weighted least-squares solution using RLS, and introduces a hardware-aware implementation to reduce activation storage and memory traffic. Experiments including pretraining at the 1.5B scale compare LDM with strong PreLN, mHC, and AttnRes baselines, demonstrating competitive language modeling quality and a distinct quality–efficiency trade-off.
Critical strengths
-
A coherent and novel synthesis of depth mixing and associative memory. The paper introduces a clean Depth Memory framework that unifies PreLN, AttnRes, and mHC through explicit read and write operations over depth. While applying sequence-mixing mechanisms along network depth has precedent in Deep Delta Learning, and recursive least squares itself is established in MesaLayer, LDM combines these ideas in a distinct way: it preserves the standard residual stream, maintains a separate fixed-size associative memory over layer representations, and recurrently maintains the exact solution to a weighted depth-regression objective. This provides a principled alternative to both explicit attention over previous layers and direct modification of the residual state.
-
Strong architectural and systems co-design. The paper identifies the activation-storage and memory-traffic costs as major bottlenecks for making a recurrent depth memory practical and addresses them through state unwinding, windowed updates, and specialized kernels. Although the measured accuracy and throughput gains over strong depth-mixing baselines are modest, the work delivers a well-integrated framework, concrete architecture, and practical implementation. This combination establishes a credible foundation for developing more capable depth-memory mechanisms in future work.
Critical weaknesses
The empirical improvements are modest relative to the added architectural and implementation complexity. Although LDM generally achieves favorable perplexity and throughput, its accuracy gains over strong alternatives such as mHC and Block AttnRes are not consistently substantial, and the main quality evaluation is limited to 400M and 1.5B models without repeated runs or uncertainty estimates. Meanwhile, LDM introduces an additional memory state, an inverse-covariance stream, specialized training logic, and hardware-specific kernels. It therefore remains unclear whether the full RLS-based design offers a sufficiently strong practical advantage over simpler depth-mixing mechanisms.
Questions for authors
Q1. Although the appendix reports peak allocated memory across training, prefill, and decoding, the paper provides no cross-method measurement of memory traffic, despite traffic reduction being one of the main contributions. Could the authors report a component-wise memory breakdown read/written for LDM, mHC, and AttnRes in each execution phase? More generally, can the authors empirically isolate the proposed representational advantage of LDM’s explicit associative memory, rather than attributing small end-to-end differences to the memory representation alone?
Q2. The quality differences among LDM, mHC, and Block AttnRes are relatively modest, while the main 400M and 1.5B results appear to rely on a single pretraining run per configuration. Were any experiments repeated with different random seeds, or can the authors provide an estimate of run-to-run variation? Evidence that the reported improvements exceed ordinary pretraining variance would materially strengthen the empirical claim.
Q3. LDM is motivated partly by the growing representation cache of AttnRes and the limited implicit memory representation of mHC. However, Block AttnRes substantially mitigates the growing-cache problem and uses less peak memory than LDM in prefill and decoding settings, while the quality differences among LDM, Block AttnRes, and mHC remain relatively small. In which practically relevant regimes does LDM provide a clear Pareto improvement in quality, throughput, and memory? A matched-resource comparison under equal peak memory, mixer-state size, or throughput would help isolate the benefit of LDM’s explicit associative memory.
4 comments
Response to Reviewer TW9R (Part 2/2)
Q2: Consistency of LDM's gains across seeds and settings
We thank you for raising this concern. We are happy to provide evidence that LDM's gains persist across settings and seeds.
First, we repeated our main comparisons (Section 4.1) at 400M scale across five random seeds, on 10B tokens to fit these runs within the rebuttal period. LDM achieves the lowest loss among all competitors on every seed. The following table shows the final validation loss of each method and seed (Table 9 in the revision, Appendix D.2):
| Seed | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|
| PreLN | 2.6951 | 2.6912 | 2.7022 | 2.6944 | 2.6921 |
| Block AttnRes | 2.6418 | 2.6365 | 2.6412 | 2.6528 | 2.6489 |
| mHC | 2.6531 | 2.6429 | 2.6506 | 2.6442 | 2.6404 |
| LDM | 2.6303 | 2.6236 | 2.6401 | 2.6280 | 2.6262 |
We have also included new results on DepthBench (Section 4.3, new in the revision), where we compare LDM against strong baselines at different width/depth aspect ratios. At 1.6B, LDM achieves the lowest loss among all reported methods at every aspect ratio (Figure 7). At 400M, it also achieves the lowest loss among all reported methods at four of the five aspect ratios, and a lower loss than Block AttnRes at every one. The following table shows the final validation loss at each depth for a fixed budget of 400M parameters (plotted as perplexity in Figure 7):
| # Layers (400M) | 16 | 20 | 24 | 28 | 32 |
|---|---|---|---|---|---|
| Block AttnRes | 2.734 | 2.715 | 2.718 | 2.726 | 2.730 |
| LDM | 2.699 | 2.684 | 2.690 | 2.696 | 2.708 |
These experiments confirm that LDM's gains persist across model shape, size and random seeds.
Q3: Matched-resource comparison
We first clarify that we describe the growing cache of AttnRes as a property of that method within our framework rather than as the motivation for LDM.
Matched FLOPs Although we acknowledge that different baselines have different peak memory overhead, our comparisons are already approximately FLOPs-matched (Table 7, Appendix C.1): all depth mixers add small, comparable matmul costs relative to PreLN. There, LDM leads (Table 2), further supported by the new experiments we discuss in our answer to Q2.
Matched memory matrix size In Section 4.1 (concretely, Table 2), we already included a comparison where we match the memory matrix size of mHC. The configuration discussed under Q1 matches the number of entries in LDM's memory matrix to mHC's expanded residual state, and it still achieves lower perplexities than mHC. It also achieves the best LAMBADA and WikiText-2 PPLs when excluding our LDM configuration.
Matched training time Although we haven't conducted specific experiments where we match the training time explicitly, we are happy to provide an analysis based on our existing 1.5B runs (Section 4.1) and the throughput of each baseline. We first measure the training throughput of the 1.5B models as in our main experiments with the protocol of Appendix C (global batch size of 1M tokens with sequences of 2K tokens on 8 B200 GPUs, median of three runs):
| Method | Tokens/s per GPU | Relative to Block AttnRes |
|---|---|---|
| PreLN | 86.1K | 1.139 |
| Block AttnRes | 75.6K | 1.000 |
| mHC | 60.3K | 0.797 |
| LDM | 68.7K | 0.909 |
In our 1.5B runs, LDM reaches Block AttnRes's final loss with 8.6% fewer tokens (Figure 4). Adjusting for wall-clock time, the token efficiency gain offsets the throughput overhead: LDM reaches Block AttnRes's final loss in about the same GPU-hours (0.5% more). However, this comparison disadvantages LDM, since it uses LDM's loss partway through the original learning rate schedule, designed for the full token budget, before the learning rate has completely decayed. Existing literature predicts that a time-matched comparison with the schedule adjusted to the available time budget would result in a lower final loss for LDM [1, Figure A1; 2, 3].
We sincerely hope that we have addressed the concerns of the reviewer satisfactorily in the revised version and would kindly ask them to update their score accordingly.
[1] Hoffmann et al. Training Compute-Optimal Large Language Models
[2] Hägele et al. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
[3] Tissue et al. Scaling Law with Learning Rate Annealing
Thank you for the effort making a thorough rebuttal that helped clarifying many aspects from my concerns. I keep my positive rating on this submission.
Thanks
Thanks for engaging with our rebuttal. We are glad many of your concerns are solved and thank you for the positive assessment.
Response to Reviewer TW9R (Part 1/2)
We thank the reviewer for appreciating the merits of our paper and for the interesting questions and suggestions. We address them below. Table and figure numbering refers to the revised manuscript, where changes are marked in blue.
Complexity We first clarify that the implementation complexity of LDM is also present in other methods. mHC changes the residual stream itself and relies on custom logic, and the same holds for AttnRes with its cache of earlier representations. Both depend on custom fused kernels and custom logic to achieve high throughput.
Regarding your question about “hardware-specific kernels”, both the mathematical formulation of LDM and our algorithmic improvements, including unwinding and windowing (Section 3.3), are independent of the GPU architecture. In our paper, we specify the hardware supported by our released kernels (Appendix F.6) because low-level implementations, which we provide for reproducibility, generally rely on hardware-specific instructions. However, as our method relies on standard primitives, it can swiftly be extended to other hardware.
Gains Aside from our main experiments, we provide new evidence that the gains are consistent. LDM achieves the lowest loss across five random seeds (400M scale, 10B tokens; Table 9 in the revision), and a lower loss than Block AttnRes at every 400M DepthBench aspect ratio (Section 4.3, new in the revision). It is also the best method in all 1.6B DepthBench runs. We report these results in detail under Q2 below.
Thank you for this question.
Memory traffic Our traffic reduction claim concerns the optimized LDM implementation relative to a naive implementation of LDM (Section 3.3). State unwinding avoids saving every memory matrix for backward, while windowed writes reduce repeated writes of that matrix: the memory matrix is written once per window instead of once per layer, which reduces the forward accesses to LDM's state at the cost of other secondary accesses. With a correct window size (E=4 in our experiments) this reduces overall traffic. Kernel fusion further reduces it.
We are happy to share the requested cross-method breakdown (Table 14 in the revision, Appendix F.4). We follow similar analytical conventions to the mHC and AttnRes papers and assume fully fused implementations. Consider the 1.5B configurations: depth blocks of eight layers for Block AttnRes, four residual streams for mHC, and Dk=8, E=4 and G=16 for LDM. Then, the forward memory traffic, i.e., the values that the depth mixing step reads and writes per token and sublayer, as a multiple of the model hidden size D (as AttnRes and mHC report traffic) is:
LDM's memory traffic is close to that of mHC despite its 2× larger state. As can be inferred in Appendix F.4, the multiplier on D stays mostly constant across scales. Table 14 additionally breaks LDM's count into its components.
Benefit of the memory representation Our ablations already isolate the contributions of the individual components of LDM (Table 3, discussed in Section 4.2). We find that keeping the residual alongside the memory is crucial in our configurations (Section 4.2(a)). Regarding the advantages of specific memory representations, this is studied in our recurrence ablations (Section 4.2(b); done at 400M scale and 15B tokens; see Table 3 for results), where we test different memory matrix updates (our main one is based on RLS, and two simpler ones, Delta and Outer). We have repeated these across five seeds to ensure consistency. The following table shows the final validation loss of each update and seed (added as Table 10 in the revision, Appendix D.2):
RLS outperforms Outer and Delta in every run, suggesting that what the memory represents (in this case, RLS corresponds to the exact solution to the depth regression problem we propose) is important.
Finally, our original submission already isolates the size of the memory itself. With Dk=4, LDM's memory matrix has the same 4D entries per token as mHC's expanded residual state. This smaller configuration still achieves lower validation, WikiText-2, and LAMBADA perplexity than mHC, alongside higher average zero-shot accuracy (Table 2, Section 4.1).