- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 2
Read review
Paper summary
This paper proposes Pyramidal Holographic Memory (PHM), a multi-resolution sequence mixer based on holographic reduced representations (HRRs). By distributing key-value bindings across local memories at different temporal scales, PHM aims to mitigate retrieval interference while retaining the computational efficiency of FFT-based operations. The paper evaluates PHM on associative recall, length extrapolation, long-range sequence benchmarks, speech classification, and language modeling, reporting stable recall at up to 64× the training sequence length and improved computational efficiency over attention at long sequence lengths.
Critical strengths
The paper provides a clear theoretical motivation for capacity-aware memory design and translates it into a relatively simple multi-scale architecture. The empirical results also demonstrate promising computational scaling, achieving a reported 7.57× speedup over SDPA at 131K tokens. These results suggest that holographic memory is a potentially useful direction for efficient long-context sequence modeling.
Critical weaknesses
-
The empirical performance of PHM remains limited compared with strong sequence-modeling baselines. In Table 3, PHM substantially underperforms S4 on ListOps and Pixel, with gaps of 18.32 and 23.75 percentage points, respectively. This raises questions about the effectiveness of PHM as a general-purpose sequence mixer, beyond its computational advantages on long sequences.
-
The 64× length extrapolation experiment provides limited evidence for general long-context retrieval capabilities. The number of relevant key-value pairs remains fixed at eight, while the additional positions are padding. Consequently, the experiment does not establish whether PHM maintains retrieval accuracy when longer sequences contain additional informative or distracting content.
-
The paper lacks sufficient ablation studies to validate its key design choices. In particular, it does not isolate the contributions of multi-scale local memories, the global memory trace, and energy-based admission. This makes it difficult to determine which components are essential to the reported improvements and whether the added architectural complexity is justified.
Questions for authors
-
Given PHM's substantial performance gap with S4 on ListOps and Pixel, how do the authors justify its effectiveness as a general-purpose sequence mixer beyond its computational advantages?
-
Can the authors evaluate length extrapolation with actual distractor tokens rather than padding to better validate PHM's long-context retrieval capabilities?
-
Can the authors provide ablations isolating the contributions of multi-scale local memories, the global memory trace, and energy-based admission?