- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the roles of operator geometry, routing, and write magnitudes in memory retrieval within a slot-routed linear associative memory. Through parameter-freezing experiments, geometric statistics, and Gaussian crosstalk modeling, the authors find that retrieval performance depends primarily on routing concentration and write magnitudes, while cross-key operator geometry remains close to random initialization. The paper also examines the applicability of these findings across different model scales and to DeltaNet.
Critical strengths
- The research question is meaningful, and the paper uses multiple ablation studies and controlled experiments to investigate the roles of different mechanisms in associative memory.
- The paper identifies the gauge dependence of conventional orthogonality metrics and attempts to establish a quantitative relationship between internal model statistics and retrieval accuracy.
Critical weaknesses
- Limited practical significance of the main findings.
Random hashing for effective addressing and the low coherence of random high-dimensional representations are already supported by prior research. The findings that fixed random routing performs comparably to learned routing and that cross-key geometry remains near its random initialization are largely consistent with existing theory and intuition. The authors also do not leverage these findings to develop a more effective memory model, limiting the practical contribution of the work.
- The operator geometry analysis does not sufficiently support broader conclusions.
The authors primarily evaluate operator geometry using Cross-key Operator Contrast, but they also find that training reduces Same-key Overlap by approximately 35%, indicating that the operator does learn useful geometric structure. Furthermore, the model already reduces cross-key interference through explicit routing, so the limited improvement in cross-key geometry may reflect the functional division within the architecture. The paper does not sufficiently disentangle these factors.
- Oversimplified experimental setting and limited generalizability.
The main experiments rely on a single-layer model with approximately 74K parameters and synthetic memory tasks. The default dataset also implicitly distinguishes keys from values through token identity. Changing this setting substantially alters the relative contributions of write magnitudes and directions. These limitations reduce the relevance of the main findings to practical sequence models.
- The DeltaNet extension does not adequately validate the core findings.
The authors find that their original static per-item attribution cannot be directly applied to DeltaNet and subsequently shift to analyzing dynamic forgetting and decoding-level behavior. Although some accuracy predictions remain effective, the main findings regarding random routing and cross-key operator geometry are not validated across architectures. This weakens the connection between the DeltaNet experiments and the paper's central conclusions.
Questions for authors
- Compared with existing theories of random addressing and associative memory, what new insights do the findings on random routing and operator geometry provide? Can these findings further guide practical model design?
- Can the authors validate their core findings on more representative linear attention architectures or real-world sequence tasks, particularly whether fixed random routing can replace learned routing?
- Which results from the DeltaNet experiments directly support the paper's central conclusions regarding operator geometry and routing?
3 comments
(Continued from the previous comment.)
- Can the authors validate their core findings on more representative linear attention architectures or real-world sequence tasks, particularly whether fixed random routing can replace learned routing?
On recall it can in three of seven published architectures and cannot in two. In the official MoM language model, learned routing lowers perplexity by at most 2.3% (Section 9, Table 3).
Recall. We trained the official implementations of seven architectures as two-layer models of about a million parameters on the Zoology recall benchmark, at ten seeds. In each we replaced the learned part that decides where a token is stored by a fixed random table of the token id, never trained. In the five architectures without a router, that part is the key computation.
| architecture | fixed minus learned, points (N = 128 / 256) | verdict |
|---|---|---|
| DeltaNet | +0.4 / +6.0 | fixed better |
| Gated DeltaNet | +2.8 / +24.3 | fixed better |
| GLA | +5.2 / +5.6 | fixed better |
| Mamba2 | +0.2 / 0.0 | not significant |
| MoM | +9.4 / +19.7 | not significant |
| linear attention (elu + 1) | −12.7 / −41.4 | fixed worse |
| Raven, 64 slots | −12.7 / −20.6 | fixed worse |
In MoM the mean favours the table, but the learned router is ahead at five of the ten seeds at N = 128 and at three at N = 256. In Mamba2 a frozen projection of the hidden state is better than the learned key (+23.1 / +10.8).
Text. In the official MoM language model (151M parameters, half a billion tokens of FineWeb-Edu, two seeds), the table in place of the router raises validation perplexity by 1.6%, with a 95% interval of 0.8 to 2.3%. Learned routing therefore helps on text, by at most 2.3%. With our memory as an extra output head of a 54M-parameter language model, a table changes perplexity by +2.4%, not significantly.
Limits of these tests (Sections 9 and 10). Every key is a single token, which favours a table. Two seeds on text cannot tell the official router from the other routers we tried. In the main comparison on text, only the table reads the previous token, and the learned router had a quarter of the official recipe's updates.
Scale and vocabulary in our memory. With sixteen times the vocabulary and four times the slots and the code size, about 4.7M parameters, and separate vocabularies, the trained router stays within 0.4 points of the fixed table in all five settings (95% intervals; Section 6). With one vocabulary at that size the fixed table scores 3.1 to 5.3 points above the fully trained model (Table 15). With one vocabulary at the main size the relative contributions do change, as you say. The write-magnitude cost is 2.3 to 5.1 points, against 14 to 27 with separate halves (a comparison chosen after its runs; the one-vocabulary runs used their own learning rates and up to six times the steps), and the direction cost is 2.0 to 6.9. The router's cost stays the largest, 15.6 to 56.7 points (Table 14).
- Which results from the DeltaNet experiments directly support the paper's central conclusions regarding operator geometry and routing?
On operator geometry, none. On routing, one, through the key computation that does the router's job in DeltaNet.
Our measurements credit each stored pair with its own contribution to the output. This per-item reading holds when a new write leaves the earlier ones unchanged. DeltaNet's delta rule first erases what the memory holds for the key, so every later key changes what an earlier write contributes. There, at one seed per load, the correct item's contribution is outranked by another item's on 84 to 95% of queries (Section 9). We had predicted that the per-item reading would hold on DeltaNet. It does not, and the paper now states that the DeltaNet results test neither the operator finding nor the diagnostic.
What DeltaNet supports is the fixed-table result: its key computation does the router's job, and a fixed table in its place is better than the learned key at both loads (first row above). The accuracy decoding of Section 7 also describes DeltaNet, within 9.8 points at one seed and 15.4 at two more, but without the split into routing and codes.
We will answer any further question within a day.
I appreciate the authors' additional cross-architecture experiments and practical model design experiments.
Both the original and additional results show that fixed random hash routing performs comparably to learned routing (with a difference of at most 1.7 accuracy points in the new experiments). However, the effectiveness of random addressing is already well supported by prior research. Therefore, I still find the novel insights offered by this paper limited and will maintain my original rating.
Thank you for the review. Section and table numbers are those of the revised PDF, now uploaded. This comment answers your first question and your point on operator geometry; the next answers questions 2 and 3.
The direction of each effect agrees with that theory. The paper adds the size of each effect in a model where the two factors separate exactly, and one design consequence, tested in a published model.
What was known. Random addresses can serve a memory (Kanerva, 2009), bindings can be stored in one fixed-width vector and read back with noise (Plate, 1995), and a random, non-orthogonal binding is near optimal for linear associative memory (Nichani et al., 2025). Section 1 credits these and the earlier comparisons of fixed with learned routers. What was not measured before, to our knowledge, is how large each effect is when where a pair is stored and how it is encoded separate exactly, and which of six released architectures admit reparametrisations of their key path: two of the six admit ones that move the customary statistic (Section 8).
What we measured. A table that is random, fills the slots equally and is never trained costs at most 1.7 points against the trained router (95% bound, ten seeds). The same router, measured against its own random initialisation, appears to be worth 37 to 54 points. The gap between these two baselines is the finding: what training adds to routing is mostly concentration and balance, and a table has both by construction. On the operator, with routing present, training raises the separation between different keys' codes by at most 6.6% in each of thirteen settings (95% upper bounds). In the 28 settings where a router is trained, the routing factor's signal-to-crosstalk ratio grows 1.4 to 1,180-fold. The mean square cosine between the queried key's codes and the other writes stays within 5.4% of its value for random codes (95% intervals; post hoc, that is, a comparison chosen after its runs; Section 8). A statistic of the codes therefore registers little of what training adds to routing.
Design. If the separation comes from routing and not from the codes, code size can be traded for slots at equal state size (slots times code size). We tested this in the official Mixture-of-Memories (MoM) code, with a fixed table in place of its router. Four memories of key size 64 recall at most 0.4 points below one memory of key size 256 at any load (95% bounds, post hoc). They need 40% fewer multiply-adds per token, and their runs are 11% shorter (Section 6, Table 28). The trade costs recall below key size 64: sixteen memories of key size 16 score below four memories at every load, by up to 14.5 points.
The revision states the first point and tests the second.
Same key. The claim concerns different keys only. Section 4 now says that what the code directions learn concerns one key: training lowers the same-key overlap by 28 to 36% at the main task's code size.
Functional division. The claim is scoped to routing being present, in the abstract and in the title of Section 4. To test your explanation we removed routing by putting all keys in one slot. If routing is why the codes do not separate, removing it should make them separate.
So your explanation holds where the codes did separate. The control does not show that the codes take over whenever routing is absent, and Section 4 says so. With routing, the rise stays small even where there is room for more: in the five routed settings where the Welch bound lets the overlap between different keys' writes fall by half or more, it is at most 2.5% (95% upper bound).