acceptodds
Under review as a conference paper at ICLR 2027

CyFA: Linear Sequence Modeling with Relative-Time-Partitioned Memory

Abstract

Linear recurrent models compress history into a fixed-size state, forcing the same finite memory to accommodate an ever-growing sequence of writes. Existing updates handle this pressure through superposition, forgetting, or replacement, which can make earlier associations difficult to recover. We introduce CyFA (Cyclic Flow Attention), which instead reuses memory by cyclically transporting the surviving history as new information arrives. Earlier writes move away from the age-zero end of the memory while new key–value pairs enter there, producing a relative-time-partitioned state in which multiple historical associations can coexist across the cycle. An exact change of coordinates reduces this moving memory to two scalar-decay linear attention recurrences, enabling efficient chunk-wise training. Across 400M–1.4B pretraining experiments, CyFA consistently outperforms linear attention models on recall-intensive tasks under matched recurrent state sizes. At 400M, it reaches on FDA versus for KDA, while maintaining strong language modeling and higher training throughput than models with fine-grained gating or Delta Rule updates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Linear Attention, Linear Sequence Modeling, Efficient Architectures

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yQtEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to improve recall in linear recurrent models with fixed-size memory. It proposes CyFA, which uses cyclic transport and a learned clock to organize historical information, followed by softmax-based retrieval. The method enables efficient training through two scalar-decay linear attention passes.

Critical strengths
  1. The paper proposes an approach to organizing finite recurrent memory through cyclic transport and a learned clock, and keeps training efficient by rewriting it as two scalar-decay linear attention passes.
  2. The proposed approach improves recall over recurrent baselines across model scales, and achieves strong results on commonsense reasoning and long-context benchmarks.
  3. The paper provides detailed derivations, useful ablations, and additional analysis of the learned clock and training stability.
Critical weaknesses
  1. Although the paper includes useful ablations, it is still unclear which design differences account for the large improvement over BLA. A more controlled comparison would help clarify the contributions of the different memory update and readout mechanisms.

2.The training throughput comparison is limited to 400M models, leaving the efficiency of CyFA at larger model scales unclear.

Questions for authors
  1. Could the authors provide a more controlled comparison with BLA to clarify which design changes contribute most to the performance improvement?
  2. Figure 5 compares training throughput only at 400M. Does CyFA maintain similar efficiency advantages over the baselines at 800M and 1.4B?
4 comments
AuthorsOpenReview ↗

Part 1/2 — W1/Q1: Controlled Comparison with BLA

Part 1/2 — W1/Q1: Controlled Comparison with BLA

We thank you for your suggestions and will add these experiments and analyses to the revised paper. Our response has two comments: Part 1 addresses W1/Q1 on BLA, and Part 2 addresses W2/Q2 on 800M/1.4B throughput.

Despite sharing the same fixed-clock Fourier–Dirichlet profile, CyFA substantially outperforms BLA. To isolate the sources of this gap, we transition from BLA to CyFA step-by-step at the 400M scale.

1. Sequence-Start Masking

Unlike CyFA, BLA masks unpopulated initial slots in the softmax. Removing this mask at ρ=2\rho=2 slightly degrades Lambada and Common but marginally improves Recall. We disable it in all subsequent comparisons to align with CyFA:

Start MaskWiki ppl ↓Lamb ppl ↓Common avg ↑Recall avg ↑
No32.9085.3639.8719.02
Yes32.8879.8640.2418.93

2. Gating Design

Both methods require state decay to prevent historical accumulation across cycles (Table 4, αt=1\alpha_t=1), but their distinct design goals lead to different gates: BLA approximates a lower-resolution SWA by coupling slot-wise erasure directly to the write profile ϕt\boldsymbol\phi_t, whereas CyFA targets a general relative-time-partitioned memory via decoupled, data-dependent scalar gates αt,βt∈(0,1)\alpha_t, \beta_t \in (0,1):

BLA: Kt=Diag⁡(1−ϕt)Kt−1+ϕtkt⊤vs.CyFA-style: Kt=αtKt−1+βtϕtkt⊤\text{BLA:} \ \mathbf K_t = \operatorname{Diag}(\mathbf 1 - \boldsymbol\phi_t)\mathbf K_{t-1} + \boldsymbol\phi_t \mathbf k_t^\top \qquad \text{vs.} \qquad \text{CyFA-style:} \ \mathbf K_t = \alpha_t \mathbf K_{t-1} + \beta_t \boldsymbol\phi_t \mathbf k_t^\top

(and identically for Vt\mathbf V_t). Replacing only BLA's gates with CyFA-style scalar gates yields a substantial improvement:

GatingClockWiki ppl ↓Lamb ppl ↓Common avg ↑Recall avg ↑
BLAρ=1\rho=133.4876.6340.6715.54
BLAρ=2\rho=232.9085.3639.8719.02
CyFA-styleρ=1\rho=126.8764.6341.8727.64
CyFA-styleρ=2\rho=226.8956.6341.2531.59

BLA's complementary gating limits performance for two main reasons:

  • Degeneration to SWA at ρ=1\rho=1: With a one-hot ϕt\boldsymbol\phi_t, each write completely overwrites its target slot, losing all recall beyond the mm-token window (which is why ρ=1\rho=1 was omitted in our paper).
  • Unconditional erasure: Because retention is dictated by the positional write profile rather than token importance, salient long-range memories are difficult to preserve across cycles.

Moreover, BLA requires two diagonal-gated GLA passes, whereas CyFA's scalar αt\alpha_t reduces to two Mamba-2-style ScalarGatedLA passes with much higher throughput (Figure 5).

3. Clock Dynamics and Readout Coordinate Frame

With CyFA-style scalar gates fixed, the clock progression and readout frame still differ:

  • Clock (δt\delta_t vs. fixed ρ\rho): BLA fixes the step size δ=1/ρ\delta = 1/\rho, whereas CyFA learns a data-dependent δt∈(0,1)\delta_t \in (0,1).
  • Readout frame (Absolute vs. Relative): BLA reads directly from absolute slots, whereas CyFA reconstructs relative-time slots via P(λt)\mathbf P(\lambda_t) before softmax. The two readouts are equivalent iff softmax⁡(P(λt)Ktqt)=P(λt)softmax⁡(Ktqt)\operatorname{softmax}(\mathbf P(\lambda_t)\mathbf K_t\mathbf q_t) = \mathbf P(\lambda_t)\operatorname{softmax}(\mathbf K_t\mathbf q_t). Integer shifts satisfy this identity as permutations, whereas fractional shifts (δ<1\delta < 1) interpolate across slots and violate it (contrary to BLA §3.2). Crucially, this non-invariance directly affects clock learning: under absolute readout, changing an early δs\delta_s shifts the absolute interpolation positions λi\lambda_i of all subsequent tokens i≥si \ge s, feeding noisy prefix gradients to δs\delta_s. Relative readout depends only on relative age λt−λi=∑u=i+1tδu\lambda_t-\lambda_i=\sum_{u=i+1}^t\delta_u (Prop. 6), canceling the common prefix λs\lambda_s and confining gradient credit to the write–readout interval.
ClockReadout FrameWiki ppl ↓Lamb ppl ↓Common avg ↑Recall avg ↑
ρ=1\rho=1Abs.26.8764.6341.8727.64
ρ=2\rho=2Abs.26.8956.6341.2531.59
Data-dependentAbs.26.7965.7341.5431.07
Data-dependentRel.26.2658.7541.8931.79

Fixed clocks (ρ=1,2\rho=1,2) exhibit a clear Common–Recall trade-off. Introducing a data-dependent clock under absolute readout brings no clear gain, whereas coupling it with relative readout reduces Wiki perplexity and achieves the best performance on both Common and Recall.

4. Learnable Slot Mixing (R\mathbf R)

Building on the data-dependent clock and relative readout, CyFA introduces a learnable matrix R\mathbf R to mix consistent relative model ages—a mechanism that relies on relative coordinates and cannot be applied directly to BLA's shifting absolute slots:

Learned R\mathbf RWiki ppl ↓Lamb ppl ↓Common avg ↑Recall avg ↑
No26.2658.7541.8931.79
Yes25.8548.1742.4733.25

Learning R\mathbf R yields further consistent gains across all metrics.

AuthorsOpenReview ↗

Part 2/2 — W2/Q2: Training Throughput at 800M and 1.4B

Part 2/2 — W2/Q2: Training Throughput at 800M and 1.4B

We extend the end-to-end training throughput benchmark in Figure 5 to the 800M and 1.4B configurations (Table 6), comparing Transformer, GDN, KDA, and full CyFA. All models are benchmarked in BF16 on a single NVIDIA RTX PRO 6000 GPU with a fixed batch budget of 32K tokens per step across sequence lengths (2K×16 to 32K×1). Throughput is reported in tokens/s:

800M

Model2K×164K×88K×416K×232K×1
Transformer58,48653,83646,16337,07825,946
GDN41,44741,31040,23737,09431,460
KDA37,01736,97335,90133,48528,879
CyFA47,47847,43047,03647,26346,702

1.4B

Model2K×164K×88K×416K×232K×1
Transformer36,98433,89130,72524,76017,507
GDN28,12527,70627,71926,19523,223
KDA25,69825,36625,32423,91821,389
CyFA31,33730,94331,36531,30930,675

Across both scales, CyFA consistently outperforms GDN and KDA at all context lengths and surpasses Transformer from 8K onward. At 32K context, CyFA achieves 1.80× / 1.48× / 1.62× the throughput of Transformer / GDN / KDA at 800M, and 1.75× / 1.32× / 1.43× at 1.4B. From 2K to 32K, CyFA retains 98.4% (800M) and 97.9% (1.4B) of its throughput.

We thank you again for these helpful suggestions and will incorporate both sets of experiments and analyses into the revised manuscript.

Thanks for the additional experiments. The controlled comparisons with BLA help clarify where the performance improvements come from, and the throughput results at 800M and 1.4B address my scalability concern. I appreciate the authors' efforts and will keep my rating at 3.

Thank you very much for taking the time to review our response and for your encouraging feedback! We are glad that the controlled comparisons with BLA and the 800M/1.4B throughput benchmarks addressed your concerns. Thank you again for your support and constructive suggestions, which have helped strengthen our paper.

Reviewer_bJETOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

CyFA uses a learned clock to shift fixed-size key–value memory through relative-time slots before each new write. An exact coordinate change turns this update into two scalar-decay attention passes for chunk-wise training. Experiments on 400M–1.4B models show improved recall at matched recurrent-state sizes and competitive language modeling. At 400M, CyFA also trains faster than most recurrent baselines.

Critical strengths
  1. The coordinate change makes cyclic transport computable with two scalar-decay attention passes. It is exact and supports efficient chunk-wise training.
  2. CyFA improves average recall over KDA by 5.74–6.94 points across three model scales under matched state and training budgets. The clock ablations also support input-dependent timing.
Critical weaknesses
  1. Table 4 tests the clock and RR, but all variants retain the relative-time readout. The BLA comparison also changes forgetting and readout (Appendix A.6). These results do not isolate the benefit of relative-time retrieval from the cyclic write addresses.
Questions for authors
  1. Does the relative-time readout improve recall over a fixed-frame readout with the same writes and gates?
1 comment
AuthorsOpenReview ↗

Isolating Relative-Time Readout and Controlled Comparison with BLA

Isolating Relative-Time Readout and Controlled Comparison with BLA

We thank you for your constructive feedback. We address W1/Q1 at the 400M scale by first isolating relative-time readout under identical cyclic writes and gates, and then tracing the controlled transition from BLA to CyFA. We will incorporate these experiments and analyses into the revised paper.

1. Direct Answer to Q1: Relative-Time vs. Fixed-Frame Readout

Holding CyFA's cyclic writes, decoupled scalar gates (αt,βt\alpha_t, \beta_t), and data-dependent clocks δt∈(0,1)\delta_t \in (0,1) fixed, we compare fixed-frame (absolute-slot) readout against relative-time reconstruction via P(λt)=ΦU(λt)Φ⊤\mathbf P(\lambda_t) = \boldsymbol\Phi\mathcal U(\lambda_t)\boldsymbol\Phi^\top before softmax:

Readout FrameLearned R\mathbf RWiki ppl ↓Lamb ppl ↓Common avg ↑Recall avg ↑
Fixed (Absolute)No (R=I\mathbf R=\mathbf I)26.7965.7341.5431.07
Relative-TimeNo (R=I\mathbf R=\mathbf I)26.2658.7541.8931.79
Relative-TimeYes25.8548.1742.4733.25

Yes—under identical writes and gates, relative-time readout improves all four metrics:

  • Direct gain at R=I\mathbf R=\mathbf I (Non-invariance & clock gradient assignment): Even without R\mathbf R, switching from fixed-frame to relative-time readout improves Recall (31.07 →\to 31.79) and reduces perplexity (26.79 →\to 26.26 on Wiki; 65.73 →\to 58.75 on Lambada). Omitting P(λt)\mathbf P(\lambda_t) leaves the output unchanged iff softmax⁡(P(λt)Ktqt)=P(λt)softmax⁡(Ktqt)\operatorname{softmax}(\mathbf P(\lambda_t)\mathbf K_t\mathbf q_t) = \mathbf P(\lambda_t)\operatorname{softmax}(\mathbf K_t\mathbf q_t). While integer shifts satisfy this identity as permutations, fractional shifts (δt<1\delta_t < 1) interpolate across multiple slots and violate it (contrary to BLA §3.2). Crucially, under fixed-frame readout, perturbing an early clock step δs\delta_s shifts the absolute interpolation coordinates λi\lambda_i of all subsequent writes i≥si \ge s, injecting noisy prefix gradients into δs\delta_s. Reconstructing the relative frame ensures that readout depends only on relative model age λt−λi=∑u=i+1tδu\lambda_t - \lambda_i = \sum_{u=i+1}^t \delta_u (Prop. 6), canceling the common prefix λs\lambda_s and confining gradient credit to the write–readout interval.
  • Enabling learnable age mixing (R\mathbf R): Because relative-time readout aligns each slot to a consistent model age at every step, it enables the learnable matrix R\mathbf R to combine relative ages, further boosting Recall to 33.25 (+2.18 points over fixed-frame readout) and lowering Lambada perplexity to 48.17. In contrast, R\mathbf R cannot be applied with a consistent temporal meaning to shifting fixed-frame coordinates.

2. Step-by-Step Controlled Transition from BLA to CyFA

To address W1, we trace each design change from BLA to CyFA at the 400M scale:

  • Step 1: Sequence-Start Masking. Unlike CyFA, BLA masks unpopulated initial slots in the softmax. Removing this mask at ρ=2\rho=2 has minimal effect (Recall 18.93 →\to 19.02, Wiki ppl 32.88 →\to 32.90), so we disable it below for a clean comparison.
  • Step 2: Gating Design (Profile-Coupled vs. Decoupled Scalar Gates). BLA couples slot-wise erasure directly to the Fourier–Dirichlet write profile ϕt\boldsymbol\phi_t (Kt=Diag⁡(1−ϕt)Kt−1+ϕtkt⊤\mathbf K_t = \operatorname{Diag}(\mathbf 1 - \boldsymbol\phi_t)\mathbf K_{t-1} + \boldsymbol\phi_t \mathbf k_t^\top), whereas CyFA uses decoupled, data-dependent scalar gates (Kt=αtKt−1+βtϕtkt⊤\mathbf K_t = \alpha_t \mathbf K_{t-1} + \beta_t \boldsymbol\phi_t \mathbf k_t^\top). Keeping fixed-frame readout and fixed clocks (ρ=1,2\rho=1, 2), replacing only BLA's gates with CyFA-style scalar gates yields a substantial gain:
GatingClockWiki ppl ↓Lamb ppl ↓Common avg ↑Recall avg ↑
BLAρ=1\rho=133.4876.6340.6715.54
BLAρ=2\rho=232.9085.3639.8719.02
CyFA-styleρ=1\rho=126.8764.6341.8727.64
CyFA-styleρ=2\rho=226.8956.6341.2531.59

BLA's coupled gating limits recall because: (i) at ρ=1\rho=1, a one-hot ϕt\boldsymbol\phi_t fully overwrites the target slot, degenerating to standard SWA with no recall beyond mm tokens; and (ii) unconditional erasure by 1−ϕt1-\boldsymbol\phi_t overwrites existing memory regardless of token importance. Moreover, scalar αt\alpha_t replaces BLA's two diagonal-gated GLA passes with two faster ScalarGatedLA passes (Figure 5).

  • Step 3: From Fixed-Clock Fixed-Frame to Full CyFA. Under fixed-frame readout, fixed clocks (ρ=1,2\rho=1, 2) trade off Common against Recall, and introducing a data-dependent clock δt\delta_t alone does not resolve this (41.54 Common, 31.07 Recall). Combining δt\delta_t with relative-time readout and learned R\mathbf R (Section 1 table) resolves this trade-off and achieves the best performance across all metrics (25.85 Wiki ppl, 48.17 Lambada ppl, 42.47 Common, 33.25 Recall).
Reviewer_oLF8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Cyclic Flow Attention (CyFA), which organizes fixed-size recurrent memory using cyclic shifts and relative-time slots. A learned, input-dependent clock controls memory movement, while scalar decay handles forgetting. The authors derive an exact coordinate transformation that reduces the memory updates to two scalar-gated linear-attention passes, enabling efficient chunk-wise training. Stop-gradient at the clock branch input is used to stabilize training.

Experiments on models ranging from 400M to 1.4B parameters show that CyFA improves recall-intensive performance over KDA under matched recurrent-state sizes, while maintaining competitive language modeling and commonsense reasoning results. CyFA also achieves better LongBench averages after continued training at 8K context length and higher training throughput than several fine-grained gating and Delta Rule baselines.

Critical strengths
  • CyFA outperforms KDA by 5.74–6.94 percentage points across 400M–1.4B models while maintaining competitive language-modeling performance (Table 2). The matched recurrent-state sizes make these gains more convincing.

  • An exact coordinate transformation reduces cyclic transport to two scalar-decay recurrences, enabling efficient chunk-wise training and higher throughput than KDA (Equations 8–11; Figure 5). Clock ablations and interventions further support the benefits of input-dependent timing (Table 4; Appendix D.2).

Critical weaknesses
  • The impact of periodic memory collisions remains unclear. With m = 127, 58.3% of heads in the 400M model have clock increments near 1, implying a cycle of roughly 127 tokens (Appendix D.2). Writes separated by a full cycle share temporal addresses, but the paper does not evaluate the resulting interference or how forgetting mitigates it. Figure 4 shows a substantial decline on MK-NIAH-1 at 8K, although the cause is unclear. This leaves CyFA's effective long-range memory capacity insufficiently characterized.

  • The gains over BLA are not sufficiently isolated. Fixed-clock CyFA and BLA share the same Fourier–Dirichlet temporal profile (Appendix A.6), but BLA is tested only at ρ = 2 and 5, while many CyFA heads have clock increments near 1. Table 4's ablations do not directly isolate differences in forgetting and readout. Comparisons at matched clock rates, with controlled changes to these mechanisms, would help establish the source of CyFA's improvements.

Questions for authors
  • How does cyclic wraparound affect practical retrieval capacity? Appendix D.2 shows that 58.3% of heads in the 400M model have clock increments near 1, implying a cycle of roughly 127 tokens. Do writes at similar cyclic positions interfere more strongly, and to what extent does forgetting mitigate this effect? A controlled analysis of clock distance and distractor count would help clarify the effective memory capacity.

  • Which design choices account for CyFA's gains over BLA? Appendix A.6 shows that both methods share the same temporal profile under a fixed clock, but BLA is tested only at ρ = 2 and 5. Could the authors compare matched clock rates and isolate the effects of forgetting and readout? Since CyFA with learned, token-independent clocks already achieves 30.32 average retrieval accuracy (Table 4), identifying the source of this gap is important for assessing the proposed mechanism.

2 comments
AuthorsOpenReview ↗

Part 1/2 — W1/Q1: Cyclic Collisions and Retrieval Capacity

Part 1/2 — W1/Q1: Cyclic Collisions and Retrieval Capacity

We thank you for your constructive comments on CyFA's memory capacity and its comparison with BLA. We have conducted additional experiments and analyses, which we will incorporate into the revised paper. We present our response in two parts: Part 1 addresses W1/Q1 on cyclic collisions and retrieval capacity, and Part 2 addresses W2/Q2 on the controlled comparison with BLA.

1. Role of forgetting in the 58.3% near-unit-clock heads (δt≈1\delta_t \approx 1)

In the 56 heads with δt≈1\delta_t \approx 1 (58.3% of the 400M model), the forget gate αt\alpha_t strongly attenuates old writes within a single 127-token cycle. Measured on 64 2K WikiText-2 sequences, the cumulative decay factor across 127 steps ∏u=t−126tαuh\prod_{u=t-126}^t \alpha_u^h has a median of 5.86×10−45.86\times 10^{-4} (retaining only 0.197% of a write's squared magnitude on average). Thus, by the time a slot address is reused in the next cycle, the previous cycle's write has already been largely forgotten.

Truncation experiments confirm that these 56 heads model local context without relying on or suffering from cross-cycle history:

  • Local language modeling (400M, 32-token continuations within 2K WikiText-2 sequences): Retaining only the most recent 127 token writes in these 56 heads leaves perplexity virtually unchanged (17.3762 →\to 17.3776), whereas clearing their memory entirely raises perplexity to 19.7346.
  • Long-range retrieval (400M, 2K MK-NIAH-1): On the 314 examples with target–question distances of at least 512 tokens, truncating history beyond 128 tokens in these 56 heads does not hurt long-range recall (36.62% →\to 36.94%). Conversely, applying the same 128-token truncation to the remaining 40 input-dependent heads (δt<1\delta_t < 1, mean δt=0.5118\delta_t = 0.5118, or ∼248\sim 248 tokens/cycle) reduces recall to 0.00%. These 40 heads learn weak decay (retaining ∼0.98\sim 0.98 amplitude over long distances) and carry the long-range associations.

2. Controlled analysis of cyclic collisions and distractor count

To test how cyclic overlap affects the long-range heads where writes persist, we run a controlled intervention on MK-NIAH-1. Keeping the text, target/query positions, and context length fixed, we relocate one distractor record so that its clock distance from the target in a primary retrieval head lands either on an exact cycle multiple (Cyclic-Aligned, Δλh mod 127≈0\Delta\lambda^h \bmod 127 \approx 0) or at symmetric offsets (Off-Phase, Δλh mod 127≈±16\Delta\lambda^h \bmod 127 \approx \pm 16):

Model & ContextKeys (Distractors)Original Recall (%)Off-Phase (%)Cyclic-Aligned (%)Diff. (Aligned − Off)
400M (2K)4 (3)40.1639.6635.14−4.52
800M (8K)4 (3)19.2020.1018.80−1.30
800M (8K)8 (7)11.8012.1011.40−0.70
  • Effect of cyclic collisions: Forcing a cyclic collision in a single head reduces recall by 4.52 points at 400M/2K and only 0.70–1.30 points at 800M/8K. Because the 40 long-range heads evolve at different, input-dependent clock rates, two writes that collide in one head remain separated in others: across the 500 MK-NIAH inputs (400M), among the 12 most retrieval-active heads, an average of only 1.30 heads experience strong target–distractor overlap on the same input.
  • Cause of the 8K drop on MK-NIAH-1 (Figure 4): At 8K, doubling the load from 4 to 8 keys reduces recall by 7.40 points (19.20% →\to 11.80%), whereas forcing cyclic alignment changes recall by only 0.70–1.30 points under both loads. This indicates that the 8K decline in Figure 4 primarily reflects fixed-state capacity pressure as more associations and background tokens accumulate (also seen in GDN and KDA in Figure 4), rather than sensitivity to specific cyclic phase collisions.
AuthorsOpenReview ↗

Part 2/2 — W2/Q2: Controlled Comparison with BLA

Part 2/2 — W2/Q2: Controlled Comparison with BLA

We thank you for suggesting a more controlled comparison with BLA. Despite sharing the same fixed-clock Fourier–Dirichlet profile, CyFA substantially outperforms BLA. To isolate the sources of this gap, we transition from BLA to CyFA step-by-step at the 400M scale.

1. Sequence-Start Masking

Unlike CyFA, BLA masks unpopulated initial slots in the softmax. Removing this mask at ρ=2\rho=2 slightly degrades Lambada and Common but marginally improves Recall. We disable it in all subsequent comparisons to align with CyFA:

Start MaskWiki ppl ↓Lamb ppl ↓Common avg ↑Recall avg ↑
No32.9085.3639.8719.02
Yes32.8879.8640.2418.93

2. Gating Design

Both methods require state decay to prevent historical accumulation across cycles (Table 4, αt=1\alpha_t=1), but their distinct design goals lead to different gates: BLA approximates a lower-resolution SWA by coupling slot-wise erasure directly to the write profile ϕt\boldsymbol\phi_t, whereas CyFA targets a general relative-time-partitioned memory via decoupled, data-dependent scalar gates αt,βt∈(0,1)\alpha_t, \beta_t \in (0,1):

BLA: Kt=Diag⁡(1−ϕt)Kt−1+ϕtkt⊤vs.CyFA-style: Kt=αtKt−1+βtϕtkt⊤\text{BLA:} \ \mathbf K_t = \operatorname{Diag}(\mathbf 1 - \boldsymbol\phi_t)\mathbf K_{t-1} + \boldsymbol\phi_t \mathbf k_t^\top \qquad \text{vs.} \qquad \text{CyFA-style:} \ \mathbf K_t = \alpha_t \mathbf K_{t-1} + \beta_t \boldsymbol\phi_t \mathbf k_t^\top

(and identically for Vt\mathbf V_t). Replacing only BLA's gates with CyFA-style scalar gates yields a substantial improvement:

GatingClockWiki ppl ↓Lamb ppl ↓Common avg ↑Recall avg ↑
BLAρ=1\rho=133.4876.6340.6715.54
BLAρ=2\rho=232.9085.3639.8719.02
CyFA-styleρ=1\rho=126.8764.6341.8727.64
CyFA-styleρ=2\rho=226.8956.6341.2531.59

BLA's complementary gating limits performance for two main reasons:

  • Degeneration to SWA at ρ=1\rho=1: With a one-hot ϕt\boldsymbol\phi_t, each write completely overwrites its target slot, losing all recall beyond the mm-token window (which is why ρ=1\rho=1 was omitted in our paper).
  • Unconditional erasure: Because retention is dictated by the positional write profile rather than token importance, salient long-range memories are difficult to preserve across cycles.

Moreover, BLA requires two diagonal-gated GLA passes, whereas CyFA's scalar αt\alpha_t reduces to two Mamba-2-style ScalarGatedLA passes with much higher throughput (Figure 5).

3. Clock Dynamics and Readout Coordinate Frame

With CyFA-style scalar gates fixed, the clock progression and readout frame still differ:

  • Clock (δt\delta_t vs. fixed ρ\rho): BLA fixes the step size δ=1/ρ\delta = 1/\rho, whereas CyFA learns a data-dependent δt∈(0,1)\delta_t \in (0,1).
  • Readout frame (Absolute vs. Relative): BLA reads directly from absolute slots, whereas CyFA reconstructs relative-time slots via P(λt)\mathbf P(\lambda_t) before softmax. The two readouts are equivalent iff softmax⁡(P(λt)Ktqt)=P(λt)softmax⁡(Ktqt)\operatorname{softmax}(\mathbf P(\lambda_t)\mathbf K_t\mathbf q_t) = \mathbf P(\lambda_t)\operatorname{softmax}(\mathbf K_t\mathbf q_t). Integer shifts satisfy this identity as permutations, whereas fractional shifts (δ<1\delta < 1) interpolate across slots and violate it (contrary to BLA §3.2). Crucially, this non-invariance directly affects clock learning: under absolute readout, changing an early δs\delta_s shifts the absolute interpolation positions λi\lambda_i of all subsequent tokens i≥si \ge s, feeding noisy prefix gradients to δs\delta_s. Relative readout depends only on relative age λt−λi=∑u=i+1tδu\lambda_t-\lambda_i=\sum_{u=i+1}^t\delta_u (Prop. 6), canceling the common prefix λs\lambda_s and confining gradient credit to the write–readout interval.
ClockReadout FrameWiki ppl ↓Lamb ppl ↓Common avg ↑Recall avg ↑
ρ=1\rho=1Abs.26.8764.6341.8727.64
ρ=2\rho=2Abs.26.8956.6341.2531.59
Data-dependentAbs.26.7965.7341.5431.07
Data-dependentRel.26.2658.7541.8931.79

Fixed clocks (ρ=1,2\rho=1,2) exhibit a clear Common–Recall trade-off. Introducing a data-dependent clock under absolute readout brings no clear gain, whereas coupling it with relative readout reduces Wiki perplexity and achieves the best performance on both Common and Recall.

4. Learnable Slot Mixing (R\mathbf R)

Building on the data-dependent clock and relative readout, CyFA introduces a learnable matrix R\mathbf R to mix consistent relative model ages—a mechanism that relies on relative coordinates and cannot be applied directly to BLA's shifting absolute slots:

Learned R\mathbf RWiki ppl ↓Lamb ppl ↓Common avg ↑Recall avg ↑
No26.2658.7541.8931.79
Yes25.8548.1742.4733.25

Learning R\mathbf R yields further consistent gains across all metrics.

We thank you again for these helpful suggestions and will incorporate both sets of experiments and analyses into the revised manuscript.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revision Description

We thank all reviewers for their constructive feedback. We revised the manuscript as follows.

  • Relative-time readout and controlled comparison with BLA (oLF8, yQtE, bJET). We added an ablation in Section 4.4 (Table 4) that isolates the effect of relative-time readout with the same write and gate designs. Appendix E.2 (Tables 11–12) presents a step-by-step controlled comparison with BLA.
  • Forgetting, head specialization, and cyclic collisions (oLF8). We expanded the analysis of clock and decay specialization in Section 4.5 (Figure 6) and added controlled retrieval experiments in Appendix D.3 (Table 9).
  • Training throughput at 800M and 1.4B (yQtE). Appendix E.3 (Figure 8) extends the end-to-end throughput benchmark to both model scales across sequence lengths from 2K to 32K.
  • Terminology and exposition. We refined terminology and exposition in the Abstract, Sections 1–3, and Section 6, and clarified the related-work discussion in Section 5.
AuthorsOpenReview ↗

Summary of Responses, Revisions, and Follow-Up

Dear Area Chair and Reviewers,

Thank you again for your thoughtful reviews and constructive suggestions. Following our earlier point-by-point responses and manuscript revision, we share this brief consolidated overview as the discussion period approaches its close, in case any final clarification would be helpful:

  • Isolating Relative-Time Readout & Controlled Comparison with BLA (Reviewers oLF8, bJET, yQtE): To isolate the readout coordinate frame under identical cyclic writes and gates (Reviewer bJET), we added a fixed-frame readout ablation in Section 4.4 (Table 4), showing that relative-time alignment improves recall (31.07→31.7931.07 \to 31.79 at R=I\mathbf{R}=\mathbf{I}, and 33.2533.25 with learned R\mathbf{R}) and perplexity by avoiding fractional-shift non-commutativity and prefix-gradient interference. In Appendix E.2 (Tables 11–12), we further trace a step-by-step transition from BLA to CyFA at matched clock rates (ρ=1,2\rho=1, 2), isolating sequence-start masking, profile-coupled vs. decoupled scalar gating (which yields the largest gain while enabling faster two-pass ScalarGatedLA execution), clock dynamics, readout coordinates, and learned slot mixing.
  • Forgetting, Head Specialization & Cyclic Collisions (Reviewer oLF8): In Section 4.5 (Figure 6) and Appendix D.3 (Table 9), we show that CyFA's 56 near-unit-clock heads (δt≈1\delta_t \approx 1) learn rapid decay (half-life <127<127 tokens; median single-cycle amplitude retention 5.86×10−45.86\times 10^{-4}), flushing previous-cycle writes before slot reuse to support local modeling without cross-cycle interference, whereas the 40 input-dependent heads carry long-range retrieval. Controlled distractor-relocation experiments on MK-NIAH-1 confirm that single-head cyclic collisions have minor impact (−4.52-4.52 pp at 400M/2K; −1.30-1.30 / −0.70-0.70 pp at 800M/8K) due to multi-head clock desynchronization, whereas doubling associative load (4 to 8 keys) at 8K lowers recall by 7.407.40 pp—suggesting that long-context multi-key retrieval is primarily limited by overall associative load within the fixed-size state rather than sensitivity to discrete periodic collisions (consistent with the 8K decline also seen for GDN and KDA in Figure 4).
  • Training Throughput at 800M and 1.4B (Reviewer yQtE): In Section 4.3 and Appendix E.3 (Figure 8), we extend our end-to-end throughput benchmark to 800M and 1.4B models across 2K–32K contexts. CyFA retains >97%>97\% of its 2K throughput at 32K, outperforms GDN and KDA at all tested lengths, and achieves 1.75×1.75\times–1.80×1.80\times Transformer throughput at 32K.

Across all three reviews, the main remaining questions centered on isolating the sources of CyFA's gains over BLA, characterizing cyclic memory collisions, and verifying training throughput at larger scales—all of which are now directly addressed with new controlled experiments and analyses in the revised paper. With only a short window remaining before the discussion period concludes, we would deeply appreciate hearing from Reviewers oLF8, bJET, and yQtE on whether our responses resolve your concerns or if any remaining questions would prevent raising your evaluation. We also hope this summary is helpful to the Area Chair in assessing how each point has been resolved.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.