- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Dissociated Recurrent Clustering (DRC), a recurrent layer that maintains a fixed-size set of weighted key/value centroids. At each step, centroids are merged to minimize clustering distortion, and the output is computed through attention over these centroids. To enable efficient training, the authors separate discrete fusion decisions from differentiable computation, allowing parallel scans once the decisions are fixed. Experiments with 1.5B hybrid models show improved loss and retrieval performance over GDN-2 at comparable state sizes.
Critical strengths
- An alternative state representation. DRC maintains individually addressable centroids rather than superposing tokens as a sum of outer products. This design achieves certain retrieval improvements over GDN-2, particularly on needle-in-a-haystack tasks.
- An interesting training strategy. The dissociation strategy enables parallel forward and backward computation once the discrete fusion decisions are fixed, allowing better GPU utilization during training.
Critical weaknesses
- Insufficient long-context evaluation. DRC's main advantage is demonstrated on RULER at 8k, but no RULER results are reported for the 32k models or beyond the training context length. Since DRC maintains a fixed number of centroids, it is important to understand how retrieval performance degrades as more tokens are compressed into each centroid. The evaluation is also limited to models up to 1.5B parameters, leaving its scalability unclear.
- Limited justification for the merging criterion. DRC minimizes weighted distortion in key space, but the extent to which this criterion controls attention approximation quality remains unclear. Although the ablation in Appendix E.3 supports key-only merging empirically, further analysis of approximation errors would strengthen the technical justification.
- Missing comparisons and ablations. The paper lacks a comparison with simple KV-cache eviction strategies, making it difficult to isolate the benefits of Ward-based merging. Comparisons with other memory and sparse-attention methods are also missing. In addition, the protected attention-sink centroid and the log-weight bias in the readout are not independently ablated.
- Efficiency benefits are not demonstrated. DRC training is 1.22× slower than the full-attention hybrid in the reported setting. Although the paper provides a detailed runtime breakdown, comparisons of inference throughput, latency, and memory usage against full attention at longer context lengths are missing. These are important for demonstrating the practical benefits of the fixed-size recurrent state.
Questions for authors
- Can you report the training time without dissociation to quantify its efficiency benefits?
- Can you explain why GDN-2 underperforms the SWA-only baseline on HumanEval+ and MBPP in Table 1, and why it outperforms DRC on several tasks in Table 2?
- Can you report RULER results for the 32k models and evaluate how retrieval performance changes beyond the training context length?
- How does the proposed Ward-based merging compare with simpler KV-cache eviction strategies? Can you also quantify its attention approximation error?
- Can you provide ablations of the protected attention-sink centroid and the log-weight bias in the readout?
- How do inference throughput, latency, and memory usage compare with full attention at longer context lengths?
Thank you for the detailed review. Your concerns are long-context evaluation, the criterion, baselines and ablations, and efficiency. The revision adds experiments on each.
Weaknesses
W1. RULER only at 8k; scale limited to 1.5B.
RULER on the 32k models up to 128k (Section 4.4, l. 478-503, Table 3): see the table in our answer to e6pa (W3).
We have launched 7B runs with matched parameters and state between DRC and GDN-2, 25% of training, DRC still strongly outperforms GDN-2 on RULER and perplexity: see our answer to e6pa (W1).
W2. Limited justification of the criterion.
See our answer to YAmu (Q2): Proposition 4 (Appendix B) and the measured output error of Table I.1.
W3. Missing eviction baselines; sink and log-weight bias not ablated.
Eviction: Appendix I (l. 1106, Table I.2). Full attention with a cache of DRC's size, under StreamingLLM, H2O or chunked SnapKV, ends 32 to 36 RULER points below DRC within 32k. (Eviction rules inside DRC: YAmu Q1.)
RULER-13 on the paper's 32k models (500 items per task), cache of 1,024 entries per KV head (DRC's state):
* Query-aware, compresses once after prefill, so it exceeds the cache during prefill.
A sparse-attention layer keeps a growing cache, like the full attention we report. Sink and bias: Appendix H (Table H.1), new runs. Without the protected sink, nothing changes (l. 127-129). Without log w in the read, the loss rises by 0.0047 and the perplexity at 8k by 0.102 [0.081, 0.129] (l. 110-111).
W4. Efficiency not demonstrated.
On the paper's 1.5B 32k models (Table 2), decode in ms per token (batch 1, one GB300):
Peak memory in GiB, same runs (about 7 GiB of it is weights and buffers):
From 8k to 128k, memory grows by 0.94 GiB for full attention against 0.12 for DRC. At the shape of the 7B model and 1M tokens, full attention needs 54.1 GiB against 23.4 for DRC (Figure L.1).
DRC stays flat from 2k to 128k. Full attention is faster at 2k and overtaken by 8k; it is 5.9 times slower at 128k. Prefill: about 1,100 tokens per second on the exact serial path, 10k to 15k on the parallel training path, against 93k to 189k for full attention up to 128k. Overall, while DRC already shows competitive decode throughput similar to that of GDN and much better than that of full attn on long sequence, we agree with the reviewer that further work is needed to realize DRC’s full FLOPs advantage at inference prefill.
Presentation (rated 1).
We will add a figure that follows one token through a fusion, the collection and the replay, and move the heavier derivations to the appendix.
Questions
Q1. Training time without dissociation.
At 1.5B scale in a 3:1 local global setup and training on 32k token sequences, dissociation reduces training step time from 3.68s to 1.44, i.e., a 2.56x gain from dissociation. Architectures with a higher ratio of DRC layers would see even greater end-to-end speedups from the dissociation.
Q2. GDN-2 below SWA on code (Table 1), above DRC in Table 2.
The MBPP score of 0.00 came from an evaluation that mishandled the recurrent cache: only 19% of the programs compiled. The corrected rerun gives 5.60, the same as the window alone, with 94.6% compiling (Tables 1 and D.1, caption of D.1). HumanEval+ has 164 problems: 1.83 against 4.88 is 3 against 8, within run-to-run variation. So on code GDN-2 is within the noise of the window, not below it. In Table 2, GDN-2 also leads full attention on TriviaQA, NQ and MMLU, so these tasks do not measure the long-range layer. The held-out code NLL gives the expected order: full attention 0.7356, DRC 0.7489, GDN-2 0.7833, window 0.9055.
Q3. RULER at 32k and beyond. See W1.
Q4. Eviction and approximation error. See W2, W3.
Q5. Sink and log-weight ablations. See W3.
Q6. Inference cost at long contexts. See W4.