Capacity-Aware Holographic Memory for Long-Context Sequence Mixing

Holographic Reduced Representations (HRRs) provide an efficient mechanism for content-addressable sequence interaction, but their associative memory suffers from interference as the number of superposed bindings grows. We show analytically that, under standard random-vector assumptions, retrieval signal-to-noise ratio scales as \(d/(N-1)\), identifying the number of bindings assigned to each memory trace as the key capacity bottleneck. Motivated by this analysis, we introduce Pyramidal Holographic Memory (PHM), a capacity-controlled multi-resolution architecture that partitions sequences across memory traces and scales, keeping the binding load per trace bounded independently of sequence length. PHM therefore preserves the computational advantages of FFT-based holographic binding while avoiding the capacity collapse of a globally overloaded trace. Across controlled associative-recall experiments, length extrapolation, long-range sequence benchmarks, speech classification, and language modeling, PHM maintains stable recall at evaluation lengths up to \(64\times\) those seen during training and increasingly outperforms optimized scaled dot-product attention at long contexts, while remaining competitive with recurrent state-space models on associative recall. These results show that explicit control of associative memory load can turn holographic binding from a capacity-limited representation into a practical primitive for long-context sequence modeling.