acceptodds
Under review as a conference paper at ICLR 2027

Capacity-Aware Holographic Memory for Long-Context Sequence Mixing

Abstract

Holographic Reduced Representations (HRRs) provide an efficient mechanism for content-addressable sequence interaction, but their associative memory suffers from interference as the number of superposed bindings grows. We show analytically that, under standard random-vector assumptions, retrieval signal-to-noise ratio scales as \(d/(N-1)\), identifying the number of bindings assigned to each memory trace as the key capacity bottleneck. Motivated by this analysis, we introduce Pyramidal Holographic Memory (PHM), a capacity-controlled multi-resolution architecture that partitions sequences across memory traces and scales, keeping the binding load per trace bounded independently of sequence length. PHM therefore preserves the computational advantages of FFT-based holographic binding while avoiding the capacity collapse of a globally overloaded trace. Across controlled associative-recall experiments, length extrapolation, long-range sequence benchmarks, speech classification, and language modeling, PHM maintains stable recall at evaluation lengths up to \(64\times\) those seen during training and increasingly outperforms optimized scaled dot-product attention at long contexts, while remaining competitive with recurrent state-space models on associative recall. These results show that explicit control of associative memory load can turn holographic binding from a capacity-limited representation into a practical primitive for long-context sequence modeling.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Holographic Reduced Representations (HRR), Sequence Mixing, Long-Context Learning, Associative Memory, Attention-Free Transformers, Efficient Sequence Modeling, FFT, Autoregressive Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pqPUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper proposes Pyramidal Holographic Memory (PHM), a multi-resolution sequence mixer based on holographic reduced representations (HRRs). By distributing key-value bindings across local memories at different temporal scales, PHM aims to mitigate retrieval interference while retaining the computational efficiency of FFT-based operations. The paper evaluates PHM on associative recall, length extrapolation, long-range sequence benchmarks, speech classification, and language modeling, reporting stable recall at up to 64× the training sequence length and improved computational efficiency over attention at long sequence lengths.

Critical strengths

The paper provides a clear theoretical motivation for capacity-aware memory design and translates it into a relatively simple multi-scale architecture. The empirical results also demonstrate promising computational scaling, achieving a reported 7.57× speedup over SDPA at 131K tokens. These results suggest that holographic memory is a potentially useful direction for efficient long-context sequence modeling.

Critical weaknesses
  1. The empirical performance of PHM remains limited compared with strong sequence-modeling baselines. In Table 3, PHM substantially underperforms S4 on ListOps and Pixel, with gaps of 18.32 and 23.75 percentage points, respectively. This raises questions about the effectiveness of PHM as a general-purpose sequence mixer, beyond its computational advantages on long sequences.

  2. The 64× length extrapolation experiment provides limited evidence for general long-context retrieval capabilities. The number of relevant key-value pairs remains fixed at eight, while the additional positions are padding. Consequently, the experiment does not establish whether PHM maintains retrieval accuracy when longer sequences contain additional informative or distracting content.

  3. The paper lacks sufficient ablation studies to validate its key design choices. In particular, it does not isolate the contributions of multi-scale local memories, the global memory trace, and energy-based admission. This makes it difficult to determine which components are essential to the reported improvements and whether the added architectural complexity is justified.

Questions for authors
  1. Given PHM's substantial performance gap with S4 on ListOps and Pixel, how do the authors justify its effectiveness as a general-purpose sequence mixer beyond its computational advantages?

  2. Can the authors evaluate length extrapolation with actual distractor tokens rather than padding to better validate PHM's long-context retrieval capabilities?

  3. Can the authors provide ablations isolating the contributions of multi-scale local memories, the global memory trace, and energy-based admission?

Reviewer_AyWMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Pyramidal Holographic Memory (PHM), a sequence-mixing architecture that organizes holographic associative memory into multiple temporal scales to control interference from superposed bindings. The method combines fourier-domain binding and retrieval with local convolution and gating. The authors provide an HRR capacity analysis and evaluate PHM on associative recall, long-range classification, speech, language modeling, length extrapolation, and computational efficiency.

Critical strengths

The paper addresses a limitation of holographic associative memory and presents an intuitive architectural approach to controlling interference. The experiments cover several sequence-modeling tasks and demonstrate favorable computational scaling relative to standard attention at long sequence lengths.

Critical weaknesses
  1. Limited novelty and concerns about the theoretical analysis.

The interference and capacity limitations of HRR are already well established [1], and HRR-based efficient sequence modeling has been investigated previously [2]. The proposed contribution primarily combines bounded local memory with multi-resolution aggregation.

Also, the SNR derivation in Appendix A.1 seems to be inconsistent with its stated assumptions. Under the stated Gaussian assumptions, the expected squared norm of each cross-talk vector is 1, rather than 1/d1/d. Hence, the claimed d/(N−1)d/(N-1) scaling doesn't really follow from the paper's definition of SNR as a ratio of total signal and noise energies.

  1. Insufficient experimental evidence for the proposed mechanism and limited empirical competitiveness.

The experiments don't sufficiently isolate the contribution of the capacity-aware hierarchy. In particular, comparisons between global-only, local-only, and multi-resolution memory configurations are missing, making it difficult to determine whether the proposed hierarchy provides benefits beyond conventional local aggregation or gating.

The length-extrapolation experiment keeps the number of relevant bindings fixed while increasing sequence length through padding. Mamba exhibits similarly stable retrieval, and full attention achieves higher accuracy. This provides limited evidence that PHM addresses interference under genuinely increasing memory demands.

Furthermore, PHM generally shows weaker performance compared to sequence model baselines on the main benchmarks, despite its favorable long-context computational scaling. Overall, the experimental results are not strong enough to establish the effectiveness or clear contribution of the proposed memory mechanism.

[1] Plate, 1995. Holographic Reduced Representations.

[2] Alam et al., 2023. Recasting Self-Attention with Holographic Reduced Representations.

Questions for authors
  1. Please clarify or correct the SNR derivation in Appendix A.1, particularly the cross-talk energy calculation, and establish how the resulting capacity analysis applies to the full multi-resolution architecture including the global trace?

  2. Please provide component ablations comparing global, local, and pyramidal memory, together with associative-recall experiments that increase the number of interfering bindings rather than merely adding padding? These would more directly test the claimed capacity-control mechanism.

Reviewer_rLM1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper analytically identifies the signal-to-noise ratio collapse in global Holographic Reduced Representations and proposes Pyramidal Holographic Memory (PHM), which partitions sequences into multi-scale, capacity-bounded local traces to enable efficient O(Nlog⁡N)O(N \log N) long-context associative retrieval.

Critical strengths
  1. The authors formally derive the SNR scaling law (SNR≈d/(N−1)SNR \approx d/(N-1)) for HRR, precisely pinpointing the mathematical root cause of capacity collapse in existing holographic mixers and directly grounding the architectural design.
  2. The LIAR experiments demonstrate that PHM maintains near-perfect recall at 32K tokens (64× its training length), proving that explicit capacity control effectively resolves the long-context generalization bottleneck.
  3. Latency benchmarks show a 7.57× speedup over optimized SDPA attention at 131K sequence length, successfully translating the theoretical sub-quadratic complexity into practical wall-clock advantages.
Critical weaknesses
  1. On LRA benchmarks requiring precise scanning or strong spatial reasoning (e.g., ListOps, Image), PHM significantly underperforms state-space models like S4, exposing the limitations of lossy compression for general sequence modeling.
  2. Due to frequent FFT executions and multi-scale memory management, PHM is empirically slower than highly optimized FlashAttention below 8K tokens, limiting its viability as a drop-in replacement for standard-length contexts.
  3. The chunk sizes SS (e.g., {32, 128, 512}) in the pyramid hierarchy appear statically configured, with no exploration of how to dynamically determine optimal capacity partitioning based on specific task distributions or sequence characteristics.
Questions for authors
  1. In autoregressive language modeling inference, how exactly does the energy-gated prefix memory decide which tokens to suppress, and could this discarding mechanism cause irreversible logical degradation during long-form generation?
  2. Since the number of fine-grained local chunks grows linearly with total sequence length, are the memory fragmentation and indexing overheads of managing these numerous independent traces masked by the current asymptotic complexity analysis in ultra-long deployments?
  3. Given that the paper primarily compares against SSMs and standard attention, why is there no direct performance and capacity comparison with recent linear-complexity linear attention variants designed for long-range dependencies, such as Gated Linear Attention (GLA) or RWKV?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.