acceptodds
Under review as a conference paper at ICLR 2027

Dissociated Recurrent Clustering

Abstract

The current status quo for large-scale recurrent models is to restrict the recurence to linear operations, which are associative and can be implemented efficiently. We depart from this key constraint with the Dissociated Recurrent Clustering (DRC), whose output is a standard attention over a recurrent state composed of a set of weighted key/value centroids that are iteratively fused to keep their number fixed. To make this process parallel during training, we introduce the “dissociation” of the computation: the centroids to fuse are computed on very large batches without autograd and, once they are fixed, the forward and the backward can be formulated as parallel scans. Experiments with hybrid 1.5B models trained on sequences of 32k tokens and evaluated on standard benchmarks, without hyper-parameter tuning, show that using DRC with 1k centroids instead of Gated DeltaNet with an equivalent recurrent state size reduces the gap to using a standard attention layer by a factor 2 in term of training loss and 3.5 in validation loss on source code, known to require long context operators.

Primary area:
generative models
Keywords:
recurrent model, associative scan, clustering, decoder transformer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zTxdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Dissociated Recurrent Clustering (DRC), a recurrent layer that maintains a fixed-size set of weighted key/value centroids. At each step, centroids are merged to minimize clustering distortion, and the output is computed through attention over these centroids. To enable efficient training, the authors separate discrete fusion decisions from differentiable computation, allowing parallel scans once the decisions are fixed. Experiments with 1.5B hybrid models show improved loss and retrieval performance over GDN-2 at comparable state sizes.

Critical strengths
  1. An alternative state representation. DRC maintains individually addressable centroids rather than superposing tokens as a sum of outer products. This design achieves certain retrieval improvements over GDN-2, particularly on needle-in-a-haystack tasks.
  2. An interesting training strategy. The dissociation strategy enables parallel forward and backward computation once the discrete fusion decisions are fixed, allowing better GPU utilization during training.
Critical weaknesses
  1. Insufficient long-context evaluation. DRC's main advantage is demonstrated on RULER at 8k, but no RULER results are reported for the 32k models or beyond the training context length. Since DRC maintains a fixed number of centroids, it is important to understand how retrieval performance degrades as more tokens are compressed into each centroid. The evaluation is also limited to models up to 1.5B parameters, leaving its scalability unclear.
  2. Limited justification for the merging criterion. DRC minimizes weighted distortion in key space, but the extent to which this criterion controls attention approximation quality remains unclear. Although the ablation in Appendix E.3 supports key-only merging empirically, further analysis of approximation errors would strengthen the technical justification.
  3. Missing comparisons and ablations. The paper lacks a comparison with simple KV-cache eviction strategies, making it difficult to isolate the benefits of Ward-based merging. Comparisons with other memory and sparse-attention methods are also missing. In addition, the protected attention-sink centroid and the log-weight bias in the readout are not independently ablated.
  4. Efficiency benefits are not demonstrated. DRC training is 1.22× slower than the full-attention hybrid in the reported setting. Although the paper provides a detailed runtime breakdown, comparisons of inference throughput, latency, and memory usage against full attention at longer context lengths are missing. These are important for demonstrating the practical benefits of the fixed-size recurrent state.
Questions for authors
  1. Can you report the training time without dissociation to quantify its efficiency benefits?
  2. Can you explain why GDN-2 underperforms the SWA-only baseline on HumanEval+ and MBPP in Table 1, and why it outperforms DRC on several tasks in Table 2?
  3. Can you report RULER results for the 32k models and evaluate how retrieval performance changes beyond the training context length?
  4. How does the proposed Ward-based merging compare with simpler KV-cache eviction strategies? Can you also quantify its attention approximation error?
  5. Can you provide ablations of the protected attention-sink centroid and the log-weight bias in the readout?
  6. How do inference throughput, latency, and memory usage compare with full attention at longer context lengths?
1 comment

Thank you for the detailed review. Your concerns are long-context evaluation, the criterion, baselines and ablations, and efficiency. The revision adds experiments on each.

Weaknesses

  • W1. RULER only at 8k; scale limited to 1.5B.
    RULER on the 32k models up to 128k (Section 4.4, l. 478-503, Table 3): see the table in our answer to e6pa (W3).

    We have launched 7B runs with matched parameters and state between DRC and GDN-2, 25% of training, DRC still strongly outperforms GDN-2 on RULER and perplexity: see our answer to e6pa (W1).

  • W2. Limited justification of the criterion.
    See our answer to YAmu (Q2): Proposition 4 (Appendix B) and the measured output error of Table I.1.

  • W3. Missing eviction baselines; sink and log-weight bias not ablated.
    Eviction: Appendix I (l. 1106, Table I.2). Full attention with a cache of DRC's size, under StreamingLLM, H2O or chunked SnapKV, ends 32 to 36 RULER points below DRC within 32k. (Eviction rules inside DRC: YAmu Q1.)

    RULER-13 on the paper's 32k models (500 items per task), cache of 1,024 entries per KV head (DRC's state):

    4k8k16k32k
    DRC60.352.843.126.0
    full attn., full cache65.560.854.542.2
    + StreamingLLM22.311.56.24.0
    + H2O20.610.35.04.3
    + chunked SnapKV24.315.49.25.5
    + one-shot SnapKV*41.333.829.522.7

    * Query-aware, compresses once after prefill, so it exceeds the cache during prefill.

    A sparse-attention layer keeps a growing cache, like the full attention we report. Sink and bias: Appendix H (Table H.1), new runs. Without the protected sink, nothing changes (l. 127-129). Without log w in the read, the loss rises by 0.0047 and the perplexity at 8k by 0.102 [0.081, 0.129] (l. 110-111).

  • W4. Efficiency not demonstrated.
    On the paper's 1.5B 32k models (Table 2), decode in ms per token (batch 1, one GB300):

    2k8k32k128k
    full attn.3.714.678.9425.81
    DRC4.464.404.404.40
    GDN-23.673.783.783.78
    SWA-1k3.493.553.553.55

    Peak memory in GiB, same runs (about 7 GiB of it is weights and buffers):

    2k8k32k128k
    full attn.6.967.007.197.94
    DRC7.937.937.958.05
    GDN-27.097.097.127.21
    SWA-1k6.956.956.987.07

    From 8k to 128k, memory grows by 0.94 GiB for full attention against 0.12 for DRC. At the shape of the 7B model and 1M tokens, full attention needs 54.1 GiB against 23.4 for DRC (Figure L.1).

    DRC stays flat from 2k to 128k. Full attention is faster at 2k and overtaken by 8k; it is 5.9 times slower at 128k. Prefill: about 1,100 tokens per second on the exact serial path, 10k to 15k on the parallel training path, against 93k to 189k for full attention up to 128k. Overall, while DRC already shows competitive decode throughput similar to that of GDN and much better than that of full attn on long sequence, we agree with the reviewer that further work is needed to realize DRC’s full FLOPs advantage at inference prefill.

  • Presentation (rated 1).
    We will add a figure that follows one token through a fusion, the collection and the replay, and move the heavier derivations to the appendix.

Questions

  • Q1. Training time without dissociation.
    At 1.5B scale in a 3:1 local global setup and training on 32k token sequences, dissociation reduces training step time from 3.68s to 1.44, i.e., a 2.56x gain from dissociation. Architectures with a higher ratio of DRC layers would see even greater end-to-end speedups from the dissociation.

  • Q2. GDN-2 below SWA on code (Table 1), above DRC in Table 2.
    The MBPP score of 0.00 came from an evaluation that mishandled the recurrent cache: only 19% of the programs compiled. The corrected rerun gives 5.60, the same as the window alone, with 94.6% compiling (Tables 1 and D.1, caption of D.1). HumanEval+ has 164 problems: 1.83 against 4.88 is 3 against 8, within run-to-run variation. So on code GDN-2 is within the noise of the window, not below it. In Table 2, GDN-2 also leads full attention on TriviaQA, NQ and MMLU, so these tasks do not measure the long-range layer. The held-out code NLL gives the expected order: full attention 0.7356, DRC 0.7489, GDN-2 0.7833, window 0.9055.

  • Q3. RULER at 32k and beyond. See W1.

  • Q4. Eviction and approximation error. See W2, W3.

  • Q5. Sink and log-weight ablations. See W3.

  • Q6. Inference cost at long contexts. See W4.

Reviewer_YAmuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dissociated Recurrent Clustering (DRC), a recurrent memory architecture that maintains a fixed number of weighted key/value centroids through online clustering. For each incoming token, DRC selects two clusters to merge according to a Ward-style clustering criterion, allowing the memory size to remain constant while preserving individually addressable representations. To enable efficient training, the authors further introduce a dissociation strategy that separates discrete fusion decisions from differentiable memory updates, allowing the latter to be computed using parallel scans. Experiments show that DRC generally outperforms existing methods, particularly on long-context retrieval tasks.

Critical strengths
  1. The idea of modeling recurrent memory as an online clustering process is interesting. Compared with conventional recurrent architectures that aggregate historical information into a hidden state, DRC maintains explicit memory representations that can be individually retrieved through attention.
  2. The dissociation strategy is technically interesting. By separating the discrete fusion decisions from the differentiable memory updates, the authors make it possible to apply parallel scans to an otherwise sequential clustering process. This idea may also be useful for other recurrent architectures involving discrete state updates.
  3. The paper includes relatively large-scale experiments with 1.5B-parameter models, and the improvements over Gated DeltaNet on long-context retrieval tasks are encouraging. The ablation studies also help explain several design choices.
Critical weaknesses
  1. The main architectural contribution of DRC is to formulate recurrent memory updates as online clustering with a predefined Ward-style fusion criterion. However, it is unclear why such a fixed clustering-based update should be preferable to learned recurrent memory updates. From a broader perspective, a matrix-valued recurrent memory can also be interpreted as a set of memory slots that are updated as new tokens arrive. The main difference is that DRC explicitly selects and merges particular slots using a predefined criterion, whereas other recurrent architectures learn how to update their hidden states.
  2. DRC selects the fusion pair that minimizes the increase in weighted clustering distortion in the key space. However, minimizing this distortion does not necessarily minimize the information loss relevant to future attention queries or downstream predictions. For example, two clusters with similar keys may contain substantially different values, yet the fusion decision does not explicitly consider this difference.
Questions for authors
  1. Have the authors considered alternative memory update rules, such as nearest-centroid updates or learned fusion mechanisms, while keeping the same memory representation and retrieval mechanism?
  2. Is there any theoretical or empirical evidence that minimizing the proposed key-space clustering distortion also minimizes the approximation error of attention outputs?
  3. How does DRC perform when the evaluation context length substantially exceeds the training length, particularly when the number of processed tokens becomes much larger than the number of centroids?
2 comments

Thank you for the positive assessment. We are glad that you find the view of memory as online clustering interesting, and the dissociation possibly useful for other recurrent models.

Weaknesses

  • W1. A fixed criterion.
    The criterion is fixed but acts on learned quantities: the keys, which decide what merges, and the weights w_t, which decide what survives. The weights matter: trained without them, the model loses 3.7 RULER points at 8k [1.9, 5.6] (Appendix K, l. 1217). Removing them at inference costs 14 to 15 points, or 36 to 38 if the fusions lose them too (Table J.3). A matrix state can be read as slots, but it reads them linearly. Hence 37.29 against 24.11 for GDN-2 on RULER at 8k (Table 1), at equal state size.

  • W2. Similar keys may hold different values.
    The read selects on keys, so a fusion is harmless when no query can separate the two clusters, which depends on keys alone. Trained with Ward on keys and values(l. 926-929): , the model has a slightly higher loss and the same RULER (Table H.1). Swapping the rule at inference changes neither RULER nor the attention error (Tables I.1, I.2). Overall, experiments indicate that computing the ward cost only on keys is both cheaper and slightly better than doing it on both keys and values.

Questions

  • Q1. Other update rules.
    Appendix I (l. 1106). Nearest centroid, the sequential k-means of MacQueen (1967), trained at 8k, costs 0.0069 of loss (Table H.1). At inference, nearest centroid, H2O and StreamingLLM lose 31 to 36 RULER points inside DRC (Table I.2). Lowest-weight eviction matches Ward within 32k and beats it by 6.5 beyond. So for retrieval, the tokens the learned weights keep matter most. However, actually merging divides the attention error by 2.7 (Table I.1) and lowers the loss.

    A fusion criterion or fusion algorithm that would itself be learned by the model is indeed a promising concept. DRC’s decision criterion is fixed, but the actual fusions do depend on the learned behavior of the model since the fusion rule is decided by learned quantities: the keys, which assess to what extent similar information is already stored in the state, and the weights w_t, which decide how important the information of each token is. These learning-based and data-dependent quantities make it so that the effective fusion that is operated is by DRC is based on how the model learns to represent tokens and weigh their importance.

    RULER-13 with the rule swapped at inference inside the paper's 32k DRC (500 items per task):

    4k8k16k32k
    Ward on keys (DRC)60.352.843.126.0
    Ward on [K,V]61.153.644.225.3
    nearest centroid19.311.910.48.0
    H2O25.313.410.39.2
    StreamingLLM19.28.95.53.5
    lowest-weight eviction59.550.840.929.3
  • Q2. Does key distortion bound the attention error?
    Proposition 4 (Appendix B, l. 689-710, pointer at l. 120-124): a centroid never overestimates the mass of its cluster. To second order the log-ratio is q^T Sigma_r q / (2 d_k). Averaged over directions and summed over clusters with their weights, it is proportional to the distortion, which the fusion increases least. Measured (Table I.1): Ward on keys has the lowest output error of six rules within 32k, and it does not grow beyond. The link is weak within a layer (Spearman 0.16), so we claim the bound, not a prediction.

  • Q3. Contexts much longer than the number of centroids.
    RULER on the 32k models up to 128k (Section 4.4 (l. 478-503), Table 3):

    4k8k16k32k64k128k
    full attn.65.560.854.542.218.21.8
    DRC60.352.843.126.016.112.6
    GDN-230.629.421.614.814.814.1

    At 128k a centroid holds 128 tokens on average, against 32 at training, and DRC keeps 12.6 on RULER where full attention drops to 1.8.

Thanks for the clarification. I have no other concern.

Reviewer_e6paOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Dissociated Recurrent Clustering (DRC), a bounded‑state recurrent layer for long‑context transformer decoders. Unlike linear‑attention / state‑space models that aggregate history as additive outer‑product sums, DRC maintains a fixed‑size set of weighted key‑value centroids. At each new token, the incoming token forms a new singleton cluster, then the pair of clusters whose fusion increases weighted Ward‑style distortion the least gets merged, keeping the total number of centroids constant. Query tokens perform softmax‑attention directly over these retained centroids. Experiments compare against full‑causal‑attention, pure sliding‑window and Gated DeltaNet (GDN‑2) baselines on 8k and 32 k sequence lengths, evaluating language perplexity, RULER long‑context retrieval, code tasks and multiple‑choice benchmarks.

Critical strengths

Instead of additive state aggregation used by linear‑attention, DRC maintains explicitly addressable weighted centroids via streaming Ward‑clustering. Forgetting is content‑driven rather than controlled by scalar gating/decay.

It cleverly separates discrete non‑parallelizable merge decisions from differentiable scan‑computed forward‑backward, enabling GPU‑accelerated training for an otherwise serial recurrent clustering mechanism.

Critical weaknesses
  1. Main experiments are confined to 470 M and 1.5 B model scales. There are no results on larger model sizes (e.g. 7B+), leaving scaling behaviour unknown. So, the model‑size scope is rather limited.

  2. Though ablations show small aggregate loss change, stale merge choices can induce unquantified distribution shift in centroid states.

  3. Evaluation mostly stays within training context window. There is little analysis of extrapolation performance to sequence lengths longer than those seen during training.

Questions for authors
  1. The dissociation mechanism re‑uses fusion decisions up to 100 optimizer steps old. Could you run controlled sweeps over cache staleness window size, to quantify how retrieval accuracy and centroid quality degrade as staleness increases?

  2. Could you contrast the conceptual differences and expected relative strengths between DRC and these competing long‑context memory designs, i.e., Mamba‑2, H2O or Compressive Transformer?

  3. Results are reported from single training runs without repeated multi‑seed replicates. Could you add multi‑seed runs and bootstrap confidence intervals to confirm that the observed performance advantages over GDN‑2 are not dominated by training randomness?

1 comment

Thank you for the review. We are glad that you see the two ideas of the paper as we do: the state is a set of addressable centroids, not an additive sum, and the content decides what is forgotten, not a gate. Your concerns are scale, staleness and extrapolation. The revision adds experiments on all three; line numbers refer to it.

Weaknesses

  • W1. Model scale.
    We started a 7B run with full attention, GDN-2 and DRC in the same configuration as the 32k family (3:1 local:global configuration). Each architecture is trained on identical batches and identical training hyper-parameters (Appendix M, l. 1235). All models are parameter matched and GDN-2 and DRC are state matched. It is not finished yet so we present intermediate results. At equal steps, DRC trails full attention by 0.004 in training loss after 52k of 120k steps, and the gap shrinks. GDN-2 trails by 0.012 after 89k steps, and the gap grows (Table M.1). RULER-13 at 25% of training (30k steps), within the 32k training length:

    4k8k16k32k
    full attn.62.951.444.735.3
    DRC63.254.844.928.1
    GDN-243.434.927.215.6

    DRC is level with or above full attention up to 16k (+3.4 at 8k) and 7.2 below at 32k. GDN-2 is 16.6 to 19.6 points below at every length.

  • W2. Stale fusions.
    Appendix E.1 quantifies the shift: with no staleness the loss is 2.1537, against 2.1540. At evaluation the weights do not change, so the fusions are exact. Q1 (below) adds a sweep with new training runs.

  • W3. Evaluation beyond the training length.
    New Section 4.4 (l. 478-503, Table 3): RULER on the 32k models up to 128k:

    4k8k16k32k64k128k
    full attn.65.560.854.542.218.21.8
    DRC60.352.843.126.016.112.6
    GDN-230.629.421.614.814.814.1

    Within 32k, DRC is 5 to 16 points below full attention and 11 to 30 above GDN-2. At 128k full attention collapses to 1.8, DRC keeps 12.6, and GDN length-extrapolates the best with 14.1.

Questions

  • Q1. Staleness sweep.
    Appendix H (l. 1063, Table H.1), new paired runs at 1.5B: recomputing the fusions every 1, 10, 30, 100 or 300 steps moves the final loss by at most 10^-4, the noise between paired runs. Staleness up to 300 steps does not hurt (l. 273).

  • Q2. Mamba-2, H2O and the Compressive Transformer.
    Mamba-2 keeps a sum of decayed outer products, read linearly, so stored items interfere. DRC writes in existing centroids knowing and optimising the resulting Ward error. The key difference is that DRC is content-aware at writing time. H2O evicts tokens and is not trained. DRC merges instead of dropping, and trains through the merges. Full attention with an H2O cache of DRC's size ends 35 RULER points below DRC within 32k (Table I.2). The Compressive Transformer compresses information with the assumption that all tokens have equal importance. DRC also compresses but it prioritizes tokens with a higher predicted weight.

  • Q3. Seeds and confidence intervals.
    Due to time and computational constraints, we could not train multiple seeds. However, the results across architectures, benchmarks, and methods are extremely consistent. Also, note that the data-loader we use assures that for a given seqlen/batch size, the data stream is deterministic, assuring that the randomization of the data does not add variance in the comparison between methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.