acceptodds
Under review as a conference paper at ICLR 2027

Separating operator geometry from learned routing in a gated, slot-routed associative memory

Abstract

Associative-memory and linear-attention architectures are often diagnosed through a functional of the binding operator: key orthogonality, the spectrum of AᵀA, a coherence bound on the feature map. We ask what training actually changes in a gated, slot-routed associative memory, in which each stored write reaches the logit through the product of a routing factor and an operator factor. Our main instrument is withheld learning: one factor is frozen at its initialisation, the rest is trained, and the task's loss is measured with every slot intact. (i) Routing. A frozen random hard hash costs 0.1 to 1.4 accuracy points (three seeds), where a frozen diffuse router costs 37 to 55: a concentrated, balanced assignment supplies most of what routing contributes. (ii) Writes. Frozen write magnitudes cost 13 to 28 points (about 5 once token identity no longer marks a write's type; two loads, one seed); frozen codebook directions cost 2 to 6. (iii) Operator geometry. Reading the trained model agrees, once the reading is a function of the model. The customary orthogonality deviation is not: it moves by up to 255× along a gauge orbit, a change of basis that leaves every logit unchanged. On a gauge-invariant contrast the operator's cross-key arrangement sits at its random-binding floor at every slot count and binding dimension we train, and no search that holds accuracy within two points lifts it past 1.03×. Within routing, concentration (how few slots a write lands on) is the lever, and arrangement (which writes collide) is a small, measurable channel: its mean square slot overlap sits a few per cent below permutations matched on concentration, and a fraction of one per cent below permutations matched on concentration and load. A relation joins the two readings. Under a Gaussian-crosstalk model whose variance assumption is checked, descriptors read off the forward pass (routing weights, write norms, same-key overlaps) predict retrieval accuracy to within 0.08 across 124 models with nothing fitted, so the reading is a measurement other scales can repeat rather than an extrapolation to them. The analysis stops where the memory forgets: on DeltaNet the static coefficient of a stored item does not exist, and the transition that removes it has a Lyapunov spectrum we measure. We do not offer a capacity law.

Primary area:
interpretability and explainable AI
Keywords:
associative memory, linear attention, routing, withheld learning, mechanistic analysis, measurement validity, gauge invariance, frame potential, mean square coherence, crosstalk model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pdHQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the roles of operator geometry, routing, and write magnitudes in memory retrieval within a slot-routed linear associative memory. Through parameter-freezing experiments, geometric statistics, and Gaussian crosstalk modeling, the authors find that retrieval performance depends primarily on routing concentration and write magnitudes, while cross-key operator geometry remains close to random initialization. The paper also examines the applicability of these findings across different model scales and to DeltaNet.

Critical strengths
  1. The research question is meaningful, and the paper uses multiple ablation studies and controlled experiments to investigate the roles of different mechanisms in associative memory.
  2. The paper identifies the gauge dependence of conventional orthogonality metrics and attempts to establish a quantitative relationship between internal model statistics and retrieval accuracy.
Critical weaknesses
  1. Limited practical significance of the main findings.

Random hashing for effective addressing and the low coherence of random high-dimensional representations are already supported by prior research. The findings that fixed random routing performs comparably to learned routing and that cross-key geometry remains near its random initialization are largely consistent with existing theory and intuition. The authors also do not leverage these findings to develop a more effective memory model, limiting the practical contribution of the work.

  1. The operator geometry analysis does not sufficiently support broader conclusions.

The authors primarily evaluate operator geometry using Cross-key Operator Contrast, but they also find that training reduces Same-key Overlap by approximately 35%, indicating that the operator does learn useful geometric structure. Furthermore, the model already reduces cross-key interference through explicit routing, so the limited improvement in cross-key geometry may reflect the functional division within the architecture. The paper does not sufficiently disentangle these factors.

  1. Oversimplified experimental setting and limited generalizability.

The main experiments rely on a single-layer model with approximately 74K parameters and synthetic memory tasks. The default dataset also implicitly distinguishes keys from values through token identity. Changing this setting substantially alters the relative contributions of write magnitudes and directions. These limitations reduce the relevance of the main findings to practical sequence models.

  1. The DeltaNet extension does not adequately validate the core findings.

The authors find that their original static per-item attribution cannot be directly applied to DeltaNet and subsequently shift to analyzing dynamic forgetting and decoding-level behavior. Although some accuracy predictions remain effective, the main findings regarding random routing and cross-key operator geometry are not validated across architectures. This weakens the connection between the DeltaNet experiments and the paper's central conclusions.

Questions for authors
  1. Compared with existing theories of random addressing and associative memory, what new insights do the findings on random routing and operator geometry provide? Can these findings further guide practical model design?
  2. Can the authors validate their core findings on more representative linear attention architectures or real-world sequence tasks, particularly whether fixed random routing can replace learned routing?
  3. Which results from the DeltaNet experiments directly support the paper's central conclusions regarding operator geometry and routing?
3 comments

Thank you for the review. Section and table numbers are those of the revised PDF, now uploaded. This comment answers your first question and your point on operator geometry; the next answers questions 2 and 3.

  1. Compared with existing theories of random addressing and associative memory, what new insights do the findings on random routing and operator geometry provide? Can these findings further guide practical model design?

The direction of each effect agrees with that theory. The paper adds the size of each effect in a model where the two factors separate exactly, and one design consequence, tested in a published model.

What was known. Random addresses can serve a memory (Kanerva, 2009), bindings can be stored in one fixed-width vector and read back with noise (Plate, 1995), and a random, non-orthogonal binding is near optimal for linear associative memory (Nichani et al., 2025). Section 1 credits these and the earlier comparisons of fixed with learned routers. What was not measured before, to our knowledge, is how large each effect is when where a pair is stored and how it is encoded separate exactly, and which of six released architectures admit reparametrisations of their key path: two of the six admit ones that move the customary statistic (Section 8).

What we measured. A table that is random, fills the slots equally and is never trained costs at most 1.7 points against the trained router (95% bound, ten seeds). The same router, measured against its own random initialisation, appears to be worth 37 to 54 points. The gap between these two baselines is the finding: what training adds to routing is mostly concentration and balance, and a table has both by construction. On the operator, with routing present, training raises the separation between different keys' codes by at most 6.6% in each of thirteen settings (95% upper bounds). In the 28 settings where a router is trained, the routing factor's signal-to-crosstalk ratio grows 1.4 to 1,180-fold. The mean square cosine between the queried key's codes and the other writes stays within 5.4% of its value for random codes (95% intervals; post hoc, that is, a comparison chosen after its runs; Section 8). A statistic of the codes therefore registers little of what training adds to routing.

Design. If the separation comes from routing and not from the codes, code size can be traded for slots at equal state size (slots times code size). We tested this in the official Mixture-of-Memories (MoM) code, with a fixed table in place of its router. Four memories of key size 64 recall at most 0.4 points below one memory of key size 256 at any load (95% bounds, post hoc). They need 40% fewer multiply-adds per token, and their runs are 11% shorter (Section 6, Table 28). The trade costs recall below key size 64: sixteen memories of key size 16 score below four memories at every load, by up to 14.5 points.

... training reduces Same-key Overlap by approximately 35%, indicating that the operator does learn useful geometric structure. Furthermore, the model already reduces cross-key interference through explicit routing, so the limited improvement in cross-key geometry may reflect the functional division within the architecture. The paper does not sufficiently disentangle these factors.

The revision states the first point and tests the second.

Same key. The claim concerns different keys only. Section 4 now says that what the code directions learn concerns one key: training lowers the same-key overlap by 28 to 36% at the main task's code size.

Functional division. The claim is scoped to routing being present, in the abstract and in the title of Section 4. To test your explanation we removed routing by putting all keys in one slot. If routing is why the codes do not separate, removing it should make them separate.

  • In four settings it did (Table 20). At vocabulary 64 and code size 64 the separation between different keys' codes rose at all three loads, by 2.8, 5.6 and 49% (95% intervals above zero; none was predicted, so these are post hoc). At 49%, accuracy was 37 points lower than with routing. It also rose by 6.8% in one of the five settings where we had predicted a rise.
  • It did not in general. Of those five settings, two fell and two did not change significantly. At the main task's vocabulary and code size the change stayed within 1.5%.

So your explanation holds where the codes did separate. The control does not show that the codes take over whenever routing is absent, and Section 4 says so. With routing, the rise stays small even where there is room for more: in the five routed settings where the Welch bound lets the overlap between different keys' writes fall by half or more, it is at most 2.5% (95% upper bound).

(Continued from the previous comment.)

  1. Can the authors validate their core findings on more representative linear attention architectures or real-world sequence tasks, particularly whether fixed random routing can replace learned routing?

On recall it can in three of seven published architectures and cannot in two. In the official MoM language model, learned routing lowers perplexity by at most 2.3% (Section 9, Table 3).

Recall. We trained the official implementations of seven architectures as two-layer models of about a million parameters on the Zoology recall benchmark, at ten seeds. In each we replaced the learned part that decides where a token is stored by a fixed random table of the token id, never trained. In the five architectures without a router, that part is the key computation.

architecturefixed minus learned, points (N = 128 / 256)verdict
DeltaNet+0.4 / +6.0fixed better
Gated DeltaNet+2.8 / +24.3fixed better
GLA+5.2 / +5.6fixed better
Mamba2+0.2 / 0.0not significant
MoM+9.4 / +19.7not significant
linear attention (elu + 1)−12.7 / −41.4fixed worse
Raven, 64 slots−12.7 / −20.6fixed worse

In MoM the mean favours the table, but the learned router is ahead at five of the ten seeds at N = 128 and at three at N = 256. In Mamba2 a frozen projection of the hidden state is better than the learned key (+23.1 / +10.8).

Text. In the official MoM language model (151M parameters, half a billion tokens of FineWeb-Edu, two seeds), the table in place of the router raises validation perplexity by 1.6%, with a 95% interval of 0.8 to 2.3%. Learned routing therefore helps on text, by at most 2.3%. With our memory as an extra output head of a 54M-parameter language model, a table changes perplexity by +2.4%, not significantly.

Limits of these tests (Sections 9 and 10). Every key is a single token, which favours a table. Two seeds on text cannot tell the official router from the other routers we tried. In the main comparison on text, only the table reads the previous token, and the learned router had a quarter of the official recipe's updates.

Scale and vocabulary in our memory. With sixteen times the vocabulary and four times the slots and the code size, about 4.7M parameters, and separate vocabularies, the trained router stays within 0.4 points of the fixed table in all five settings (95% intervals; Section 6). With one vocabulary at that size the fixed table scores 3.1 to 5.3 points above the fully trained model (Table 15). With one vocabulary at the main size the relative contributions do change, as you say. The write-magnitude cost is 2.3 to 5.1 points, against 14 to 27 with separate halves (a comparison chosen after its runs; the one-vocabulary runs used their own learning rates and up to six times the steps), and the direction cost is 2.0 to 6.9. The router's cost stays the largest, 15.6 to 56.7 points (Table 14).

  1. Which results from the DeltaNet experiments directly support the paper's central conclusions regarding operator geometry and routing?

On operator geometry, none. On routing, one, through the key computation that does the router's job in DeltaNet.

Our measurements credit each stored pair with its own contribution to the output. This per-item reading holds when a new write leaves the earlier ones unchanged. DeltaNet's delta rule first erases what the memory holds for the key, so every later key changes what an earlier write contributes. There, at one seed per load, the correct item's contribution is outranked by another item's on 84 to 95% of queries (Section 9). We had predicted that the per-item reading would hold on DeltaNet. It does not, and the paper now states that the DeltaNet results test neither the operator finding nor the diagnostic.

What DeltaNet supports is the fixed-table result: its key computation does the router's job, and a fixed table in its place is better than the learned key at both loads (first row above). The accuracy decoding of Section 7 also describes DeltaNet, within 9.8 points at one seed and 15.4 at two more, but without the split into routing and codes.

We will answer any further question within a day.

I appreciate the authors' additional cross-architecture experiments and practical model design experiments.

Both the original and additional results show that fixed random hash routing performs comparably to learned routing (with a difference of at most 1.7 accuracy points in the new experiments). However, the effectiveness of random addressing is already well supported by prior research. Therefore, I still find the novel insights offered by this paper limited and will maintain my original rating.

Reviewer_kLdeOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks what training actually changes in a small associative memory the authors build themselves (73,728 parameters, one block). In this model, each stored item reaches the output as the product of a routing factor (which slots it is written to) and an operator factor (the learned key/value binding). The main finding is withheld learning: freeze one part at its random initialization, train the rest, and measure the accuracy lost on a synthetic recall task. Freezing the router at a diffuse initialization costs 37–55 points, but a fixed random balanced hash costs only 0.1–1.4, so routing mostly needs concentration and balance rather than learning. Write magnitudes are worth 13–28 points and codebook directions 2–6. The paper also shows that the usual key-orthogonality diagnostic changes 255× under a change of basis that leaves the model’s outputs unchanged, while gauge-invariant measures show the operator staying at its random baseline. A simple crosstalk model predicts accuracy within 0.08 across 124 models with nothing fitted.

Critical strengths
  • The experiments are careful and well controlled. Withheld learning directly measures what each part contributes, every comparison has a matched baseline, and the gauge argument convincingly shows that a common diagnostic is not a property of the model.
  • The paper is unusually transparent: the failed DeltaNet control is reported, a full record of corrections is included.
Critical weaknesses
  • The scope is limited to one small custom model (73K parameters, a single memory layer) and one synthetic recall task. This is reasonable for a controlled study, but the paper’s critique targets diagnostics used for real linear-attention models, and these diagnostics are never re-checked on those models, so it is unclear how far the claims carry. Some findings also depend on how the task is built: keys and values come from separate halves of the vocabulary, so the model can tell a write’s type from the token itself. When keys and values share one vocabulary, this write gate disappears, the write-magnitude cost drops from 13–20 to about 5 points, and two of the four loads do not converge at all.
  • The paper is very difficult to follow. The writing is dense, relies on many undefined or nonstandard terms (gauge, floor, arrangement, fingerprint), packs several numbers and caveats into each sentence, and spreads about ten loosely connected analyses across 25 pages of appendices, so the main contribution is hard to identify. The paper discloses that an LLM drafted the prose; however it was produced, the text needs substantial rewriting for clarity before it can be properly evaluated by a general ICLR audience.
Questions for authors
  • Against a fixed hash, learned routing adds only 0.1–1.4 points (above seed spread only at N = 48). Is the real finding that routing needs concentration and balance, not learning? The diffuse router’s cost also depends on its untuned initial scale.
  • Most of the 255× gauge effect comes from simple scalar rescaling; across non-scalar gauges it is only 5.5×. Many real models (e.g., DeltaNet) normalize keys, which removes this freedom. Does the gauge problem matter in published architectures?
  • How much of the write-magnitude result survives with a shared vocabulary if training is tuned so all loads converge?
  • After load matching, the arrangement effect is only 0.2–0.6%, costing 0.001–0.011 accuracy. Is it practically meaningful?
  • Several analyses use a single seed (ceiling, learning-rate sweep, ladder, M/n sweep, shared vocabulary). Can you report more seeds to verify the results?
2 comments

Thank you for the careful review. Section and table numbers are those of the revised PDF, now uploaded. This comment answers your points on published models and on routing; the next answers questions 3 to 5 and the writing.

... the paper's critique targets diagnostics used for real linear-attention models, and these diagnostics are never re-checked on those models, so it is unclear how far the claims carry.

  1. Most of the 255× gauge effect comes from simple scalar rescaling; across non-scalar gauges it is only 5.5×. Many real models (e.g., DeltaNet) normalize keys, which removes this freedom. Does the gauge problem matter in published architectures?

It matters in two of six released 1.3B-parameter architectures, GLA and RetNet, and not in the other four, DeltaNet among them.

For each architecture we derived from its released code which changes of the key coordinates leave every output unchanged, and checked them numerically on the released checkpoint (Section 8, Table 2).

  • In GLA and RetNet such changes move the customary statistic, the mean absolute cosine between different tokens' keys. Random ones move it by +10% on GLA and +41% on RetNet. A search finds movements up to 3.7-fold when every scale stays within a factor 1.73 of one.
  • Under that same bound, such a change can reverse the statistic's order between two layers. One exists for each of the 10 pairs of five layers on RetNet and for 7 of them on GLA. A layer-to-layer trend in the statistic is therefore not a property of those two models.
  • In Mamba2, HGRN2, DeltaNet and Gated DeltaNet only permutations of the key coordinates pass, and they leave the cosine fixed. There the statistic is a property of the model, as you expected for DeltaNet.

You are right about the 255×. It comes from a uniform rescaling, and key normalisation removes that rescaling; the abstract no longer cites the number. In our memory, changes that are neither a rotation nor a uniform rescaling move the statistic 5.2 to 6.2-fold on four checkpoints (Section 2.3).

Where the statistic is a property of the model, its value still does not by itself indicate more or less learning. At the middle layer the trained keys are more separated than under a random key map of the same shape in three of those four architectures (0.39 to 0.83 times the random value). In Gated DeltaNet they are more alike (1.41 times; Table 7).

  1. Against a fixed hash, learned routing adds only 0.1–1.4 points (above seed spread only at N = 48). Is the real finding that routing needs concentration and balance, not learning? The diffuse router's cost also depends on its untuned initial scale.

Yes, with a remainder of at most 1.7 points, and the revision states the finding in your terms.

  • At ten seeds the trained router is better than the fixed table at the two higher loads, by 1.4 and 0.7 points on average and by at most 1.7 (95% bound). At the two lower loads the difference is not significant and at most 0.8 (Table 9).
  • The diffuse router's cost does depend on its initial scale. It is 37 to 54 points at the scale we used (Table 1). Doubling the initial scores lowers it to 5 to 21 points, and halving them raises it to 65 to 79 (Table 10).

Section 1 now reads: "What training adds to routing is mostly concentration (few slots per token) and balance (evenly filled slots; Section 3)." The 37 to 54 points are reported as what the router appears to be worth when measured against its own random initialisation, not as what learning adds.

One note on your summary. The crosstalk model takes its inputs from each model's own forward pass, so it decodes accuracy and does not forecast it. Its mean absolute error is 2.8 points over 124 models (Section 7). A separate forecast written before training missed sixty further models by 3.1 points on average.

(Continued from the previous comment.)

  1. How much of the write-magnitude result survives with a shared vocabulary if training is tuned so all loads converge?

2.3 to 5.1 points survive, against 14 to 27 with separate halves. We selected the learning rate per variant and load and trained up to six times as long, at three seeds (Table 14). Even so, 15 of the 60 runs did not converge by our criterion, and the costs include them. At N = 64, where 14 of the 15 runs converged, the cost is 2.4 points (95% interval 2.1 to 2.7). The comparison with separate halves was chosen after the runs, and the paper labels it post hoc. The abstract now says that this effect shrinks when keys and values share one vocabulary. With one vocabulary, freezing the router still costs the most, 15.6 to 56.7 points. The separation between different keys' codes stays within −7.5 and +5.8% of its value before training (95% intervals, none significant, also post hoc; Appendix G).

  1. After load matching, the arrangement effect is only 0.2–0.6%, costing 0.001–0.011 accuracy. Is it practically meaningful?

As a measurement, yes: it is the third of three parts of routing. It is worth up to about two points in the trained router, with the codes retrained, and 9.3 to 16.6 points in tables built to isolate it (Sections 3 and 5).

  • Concentration: the router held at its diffuse initialisation costs 37 to 54 points.
  • Balance: a table that fills the slots unevenly loses 8 to 21 points against one that fills them equally.
  • Which tokens share a slot: permuting each token's slots, which also changes the balance, and retraining the codes costs 1.9 and 2.0 points at the two higher loads. At the two lower loads the cost is not significant and at most 1.5 (95% bound).

Your numbers are the third part read without retraining, at one seed: the fifty load-matched assignments cost 0.1 to 1.1 points (Table 22). Two tables built to isolate this part, with the same concentration and the same mean overlap, differ by 9.3 to 16.6 points, depending only on which tokens' rows overlap. The trained router uses little of this part. For a designer this means that concentration and balance come first.

  1. Several analyses use a single seed (ceiling, learning-rate sweep, ladder, M/n sweep, shared vocabulary). Can you report more seeds to verify the results?

All five now have three seeds, except the search without an accuracy limit in the ceiling analysis. The comparison of the fixed table with the trained router has ten.

  • Ceiling: among searched codebooks that lose at most two points of accuracy, the gain in the separation between different keys' codes is at most 5.3% (95% upper bound, three seeds; Table 19). The search without the accuracy limit is still at one seed, and the paper says so.
  • Learning-rate sweep: no cost shifts significantly between the main experiments' rate and the selected one. The shifts' 95% intervals lie within −4.4 and +2.8 points (Appendix F; Table 12 gives the costs at the selected rates).
  • Ladder: accuracy falls as the overlap rises in 11 of the 16 steps between neighbouring tables, and accuracy rises in 2 (post hoc). No single run's rank correlation is significant after correction for the twelve runs, and the paper now says so (Section 5, Table 13).
  • M/n sweep: the separation between different keys' codes stays within −7.7 and +8.1% of its value before training (95% intervals, none significant; Section 6, Table 18).
  • Shared vocabulary: answered above; the write-magnitude result shrinks.

The paper is very difficult to follow. The writing is dense, relies on many undefined or nonstandard terms (gauge, floor, arrangement, fingerprint), packs several numbers and caveats into each sentence, and spreads about ten loosely connected analyses across 25 pages of appendices, so the main contribution is hard to identify. ...

The main text is rewritten. The introduction states the thesis and three findings before any notation. The four terms you list are gone as terms. A gauge change, for example, is now a reparametrisation, introduced as "changes of parameters that leave every output unchanged", and the arrangement effect is described as which tokens share a slot. Section 2 defines the memory's terms and statistics before the results use them. Section 2.2 states once how every comparison is judged. Table 5 maps each section of the main text to the appendices that test its claims.

We will answer any further question within a day.

Reviewer_VzmtOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper asks, in a purpose-built gated slot-routed associative memory (73,728 parameters, V=512, M=16 slots, n=8 binding dimension, trained on the official Zoology-style recall generator), which factors training actually changes when each stored write reaches the logit as (routing factor) times (operator factor) — an exact readout identity (Eq. 1) the authors verify numerically to 1e-15. The method is "withheld learning": freeze one factor at initialisation, train the rest, measure the cost.

Critical strengths

The paper is very honest about the LLM usage.

Critical weaknesses

Generality: Every quantitative claim comes from a single custom-built memory (disclosed as the reduction of an unpublished larger model) on one synthetic associative-recall generator at one load ray.

Presentation: The paper is very difficult to follow. Concepts are used before they were introduced to the reader.

Questions for authors

What's the conclusion of the experiments? What's the thesis of the paper?

2 comments

Thank you for the review. Section and table numbers are those of the revised PDF, now uploaded.

What's the conclusion of the experiments? What's the thesis of the paper?

In this memory, the separation between different keys, beyond what random codes already give, comes from routing, and the customary key-orthogonality statistic cannot show this.

The memory stores key–value pairs. Each stored pair reaches the output as an exact product of two factors: where the pair is stored (a router chooses among sixteen slots) and how it is encoded (a learned code for the key and one for the value). To see which factor keeps different keys apart, we hold one factor at its random initialisation, train the rest, and measure the accuracy lost. The experiments conclude three things.

  • Routing. A random table that fills the slots equally and is never trained costs at most 1.7 accuracy points against the trained router (95% bound, ten seeds). The router held at its random initialisation costs 37 to 54 points. What training adds to routing is therefore mostly concentration (few slots per token) and balance (evenly filled slots), and a table has both by construction.
  • Write magnitudes. Training learns how strongly to write each pair. Freezing these magnitudes costs 14 to 27 points. It costs 2.3 to 5.1 when keys and values share one vocabulary and training is tuned to it (a comparison chosen after its runs).
  • Codes. With every write scaled to unit length, training raises the separation between different keys' codes by at most 6.6% in each of thirteen routed settings (95% upper bounds).

The customary statistic is computed from the key codes alone, so it cannot register what training added to routing. It also moves under changes of parameters that leave every output unchanged, in our memory and in two of six released architectures. The abstract and Section 1 now state this thesis and these findings before any notation.

Generality: Every quantitative claim comes from a single custom-built memory (disclosed as the reduction of an unpublished larger model) on one synthetic associative-recall generator at one load ray.

That was true of the submission's main findings. The revision tests them outside that memory and reports where they stop (Section 9, Table 3).

  • Larger versions of the memory. With sixteen times the vocabulary and four times the slots and the code size, about 4.7M parameters, and separate vocabularies for keys and values, the trained router stays within 0.4 points of the fixed table in all five settings (95% intervals; Section 6).
  • Published architectures. In seven published architectures, trained on the Zoology recall benchmark at ten seeds, we replace the learned part that decides where a token is stored by a fixed table. The table is better in three (DeltaNet, Gated DeltaNet, GLA), worse in two (the original linear attention, Raven) and not significantly different in two (Mamba2, Mixture-of-Memories).
  • Released models. In two of six released 1.3B-parameter architectures, GLA and RetNet, the statistic can be moved without changing any output (Section 8, Table 2).
  • Text. The recall tests all use the Zoology generator; two further tests use text. In the official Mixture-of-Memories language model (151M parameters), a fixed table in place of the router raises perplexity by 1.6%, so learned routing lowers it by at most 2.3% (95% bound, two seeds).

The finding on the codes is tested only in our memory and its larger versions (Sections 4, 6 and 10). The clause about a larger model is deleted, since no result in the paper uses that model: Section 2.1 describes the memory as our own construction, not a published architecture.

Presentation: The paper is very difficult to follow. Concepts are used before they were introduced to the reader.

The main text is rewritten so that Section 2 introduces the memory's concepts before the results use them. The introduction states the thesis and the findings in plain words, as above. Sections 2.1 and 2.2 then define the memory, the three numbers through which each stored pair enters a read, and the statistics, before the results that use them. Section 2.2 states once how every comparison is judged. Table 5 maps each section to the appendices that test its claims.

We will answer any further question within a day.

I thank the authors for the response and the revision. However, I still don't see how current results could add to the associative memory literature. Therefore, I maintain my original score.

Comments on the paper

1 comment

We thank the three reviewers. The revised PDF is now uploaded; section and table numbers below are those of the revision.

The method and the thesis are those of the submission. Item 4 below states a result that shrank, and item 7 the rules by which every comparison is now judged. In one sentence: in a routed associative memory, the separation between different keys, beyond what random codes already give, comes from routing, and the customary key-orthogonality statistic cannot show this. The revision rewrites the main text for reading order and adds the tests below.

  1. Reading order (Vzmt, kLde). The abstract and the introduction state the thesis and three findings in plain words before any notation. Section 2 then defines the memory and its statistics before the results use them. Table 5 maps each section to the appendices that test its claims.
  2. Tests outside the small memory (all three reviewers). The key-orthogonality statistic is checked on six released 1.3B-parameter architectures, and output-preserving reparametrisations move it in two of the six (Section 8, Table 2). A fixed table replaces the learned router or key computation in seven published architectures on Zoology recall, at ten seeds, and in two tests on text (Section 9, Table 3). The table is better in three of the seven and worse in two; in the official MoM language model, learned routing lowers perplexity by at most 2.3% (95% bound, two seeds). Larger versions of our memory reach about 4.7M parameters (Sections 6 and 10).
  3. More seeds (kLde). The five analyses that had one seed now have three (one search in the ceiling analysis stays at one seed), and the comparison of the fixed table with the trained router has ten (Appendices F to H).
  4. One vocabulary for keys and values (kLde, pdHQ). Tested at two sizes. The write-magnitude result shrinks, and the abstract now says so (Sections 3 and 6, Appendix G).
  5. A control without routing (pdHQ). With all keys in one slot, it tests whether the codes separate once routing is removed. In four of the thirteen settings the 95% interval of the change in their separation lies above zero, and in two it lies below (Section 4, Table 20).
  6. One design consequence (pdHQ). At equal state size, code size can be traded for slots. In the official Mixture-of-Memories code, with a fixed table in place of its router, four memories of key size 64 recall at most 0.4 points below one memory of key size 256 (95% bounds; post hoc, that is, a comparison chosen after its runs) and need 40% fewer multiply-adds (Section 6).
  7. How comparisons are judged. Every comparison is a mean difference with a 95% interval. At ten seeds it is decided by the share of seeds won (Bouthillier et al., 2021), and at fewer by a paired t-test with Holm's correction (Section 2.2). We adopted these rules after the runs, in place of thresholds of our own. Table 6 lists the fourteen main-text claims whose verdict differs under the earlier rules.
  8. Scope and limits. The abstract and Section 1 state the scope: routed associative memories that do not forget, on recall with single-token keys. Section 9 reports where the findings stop, and Section 10 lists the limits.
  9. What was known, and what is new. Section 1 separates what was known from what is measured here. What was not measured before, to our knowledge, is how large each effect is when where a pair is stored and how it is encoded separate exactly, and which of six released architectures admit reparametrisations of their key path. Section 1 credits the tensor-product readout (Smolensky, 1990), random addressing (Kanerva, 2009) and the storage of bindings in one fixed-width vector (Plate, 1995). It also credits fixed and frozen routers in mixture-of-experts layers (Roller et al., 2021; Fan et al., 2024) and work on cosines that move while a product stays the same (Steck et al., 2024; Cain, 2026).

We answer each review in its own thread.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.