- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies how to improve recall in linear recurrent models with fixed-size memory. It proposes CyFA, which uses cyclic transport and a learned clock to organize historical information, followed by softmax-based retrieval. The method enables efficient training through two scalar-decay linear attention passes.
Critical strengths
- The paper proposes an approach to organizing finite recurrent memory through cyclic transport and a learned clock, and keeps training efficient by rewriting it as two scalar-decay linear attention passes.
- The proposed approach improves recall over recurrent baselines across model scales, and achieves strong results on commonsense reasoning and long-context benchmarks.
- The paper provides detailed derivations, useful ablations, and additional analysis of the learned clock and training stability.
Critical weaknesses
- Although the paper includes useful ablations, it is still unclear which design differences account for the large improvement over BLA. A more controlled comparison would help clarify the contributions of the different memory update and readout mechanisms.
2.The training throughput comparison is limited to 400M models, leaving the efficiency of CyFA at larger model scales unclear.
Questions for authors
- Could the authors provide a more controlled comparison with BLA to clarify which design changes contribute most to the performance improvement?
- Figure 5 compares training throughput only at 400M. Does CyFA maintain similar efficiency advantages over the baselines at 800M and 1.4B?
4 comments
Part 2/2 — W2/Q2: Training Throughput at 800M and 1.4B
Part 2/2 — W2/Q2: Training Throughput at 800M and 1.4B
We extend the end-to-end training throughput benchmark in Figure 5 to the 800M and 1.4B configurations (Table 6), comparing Transformer, GDN, KDA, and full CyFA. All models are benchmarked in BF16 on a single NVIDIA RTX PRO 6000 GPU with a fixed batch budget of 32K tokens per step across sequence lengths (2K×16 to 32K×1). Throughput is reported in tokens/s:
800M
| Model | 2K×16 | 4K×8 | 8K×4 | 16K×2 | 32K×1 |
|---|---|---|---|---|---|
| Transformer | 58,486 | 53,836 | 46,163 | 37,078 | 25,946 |
| GDN | 41,447 | 41,310 | 40,237 | 37,094 | 31,460 |
| KDA | 37,017 | 36,973 | 35,901 | 33,485 | 28,879 |
| CyFA | 47,478 | 47,430 | 47,036 | 47,263 | 46,702 |
1.4B
| Model | 2K×16 | 4K×8 | 8K×4 | 16K×2 | 32K×1 |
|---|---|---|---|---|---|
| Transformer | 36,984 | 33,891 | 30,725 | 24,760 | 17,507 |
| GDN | 28,125 | 27,706 | 27,719 | 26,195 | 23,223 |
| KDA | 25,698 | 25,366 | 25,324 | 23,918 | 21,389 |
| CyFA | 31,337 | 30,943 | 31,365 | 31,309 | 30,675 |
Across both scales, CyFA consistently outperforms GDN and KDA at all context lengths and surpasses Transformer from 8K onward. At 32K context, CyFA achieves 1.80× / 1.48× / 1.62× the throughput of Transformer / GDN / KDA at 800M, and 1.75× / 1.32× / 1.43× at 1.4B. From 2K to 32K, CyFA retains 98.4% (800M) and 97.9% (1.4B) of its throughput.
We thank you again for these helpful suggestions and will incorporate both sets of experiments and analyses into the revised manuscript.
Thanks for the additional experiments. The controlled comparisons with BLA help clarify where the performance improvements come from, and the throughput results at 800M and 1.4B address my scalability concern. I appreciate the authors' efforts and will keep my rating at 3.
Thank you very much for taking the time to review our response and for your encouraging feedback! We are glad that the controlled comparisons with BLA and the 800M/1.4B throughput benchmarks addressed your concerns. Thank you again for your support and constructive suggestions, which have helped strengthen our paper.
Part 1/2 — W1/Q1: Controlled Comparison with BLA
Part 1/2 — W1/Q1: Controlled Comparison with BLA
We thank you for your suggestions and will add these experiments and analyses to the revised paper. Our response has two comments: Part 1 addresses W1/Q1 on BLA, and Part 2 addresses W2/Q2 on 800M/1.4B throughput.
Despite sharing the same fixed-clock Fourier–Dirichlet profile, CyFA substantially outperforms BLA. To isolate the sources of this gap, we transition from BLA to CyFA step-by-step at the 400M scale.
1. Sequence-Start Masking
Unlike CyFA, BLA masks unpopulated initial slots in the softmax. Removing this mask at ρ=2 slightly degrades Lambada and Common but marginally improves Recall. We disable it in all subsequent comparisons to align with CyFA:
2. Gating Design
Both methods require state decay to prevent historical accumulation across cycles (Table 4, αt=1), but their distinct design goals lead to different gates: BLA approximates a lower-resolution SWA by coupling slot-wise erasure directly to the write profile ϕt, whereas CyFA targets a general relative-time-partitioned memory via decoupled, data-dependent scalar gates αt,βt∈(0,1):
BLA: Kt=Diag(1−ϕt)Kt−1+ϕtkt⊤vs.CyFA-style: Kt=αtKt−1+βtϕtkt⊤
(and identically for Vt). Replacing only BLA's gates with CyFA-style scalar gates yields a substantial improvement:
BLA's complementary gating limits performance for two main reasons:
Moreover, BLA requires two diagonal-gated GLA passes, whereas CyFA's scalar αt reduces to two Mamba-2-style
ScalarGatedLApasses with much higher throughput (Figure 5).3. Clock Dynamics and Readout Coordinate Frame
With CyFA-style scalar gates fixed, the clock progression and readout frame still differ:
Fixed clocks (ρ=1,2) exhibit a clear Common–Recall trade-off. Introducing a data-dependent clock under absolute readout brings no clear gain, whereas coupling it with relative readout reduces Wiki perplexity and achieves the best performance on both Common and Recall.
4. Learnable Slot Mixing (R)
Building on the data-dependent clock and relative readout, CyFA introduces a learnable matrix R to mix consistent relative model ages—a mechanism that relies on relative coordinates and cannot be applied directly to BLA's shifting absolute slots:
Learning R yields further consistent gains across all metrics.