acceptodds
Under review as a conference paper at ICLR 2027

PAY AT THE EVENT: EVENT-CONDITIONED RECURRENT STATE TRACKING AND ASSOCIATIVE RECALL

Abstract

We introduce an event-conditioned recurrent block that explicitly separates transition computation from associative storage. FLINT maintains a low-cost diagonal carry at every token while applying expressive Householder transformations only at selected events. When event gates are determined before scanning, the recurrent dependency chain can be reorganized around event boundaries, making expensive transition work proportional to the number of events. EGAM provides fixed-size associative memory through binary addressing, gated additive writes, and normalized slot reads. In a shared-backbone block benchmark, the combined block achieves faster forward execution than Gated DeltaNet-2 at known-mask density and length . Replacing memory-projection multiplications with additive and bitwise operations reduces mixer multiply-accumulates (MACs) from M to M per token and layer, with M accumulate/bit operations, excluding the MLP. Across three seeds, mean WikiText-103 perplexity changes from to . On FineWeb-Edu, a 122.54M-parameter model trained on 1.007B tokens retains validation perplexity ( to , one seed) while reducing mixer MACs from M to M. The combined task state is independent of context length and, at length , is approximately one-eighth the size of attention’s KV cache. Learned-gate and joint tasks evaluate tracking and recall within these budgets. We will release our code upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
event-based sequence modeling, efficient sequence modeling, associative recall

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Zt9fOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes FLINT, an event-conditioned recurrent state tracker, and EGAM, a fixed-capacity associative memory, to reduce state-transition computation and memory-projection costs while preserving language modeling quality.

Critical strengths
  1. The paper presents a principled decomposition of recurrent state tracking and associative recall, addressing complementary limitations of efficient sequence models.
  2. FLINT introduces an exact event-boundary execution schedule that reduces expensive state-transition work in proportion to the number of events.
  3. Experiments demonstrate substantial reductions in mixer MACs and estimated arithmetic energy with minimal perplexity degradation across two language modeling benchmarks.
Critical weaknesses
  1. The language modeling experiments use relatively small models and limited training budgets, with only one seed reported for the FineWeb-Edu evaluation.
  2. The arithmetic energy improvements are based on an analytical cost model rather than direct hardware energy measurements, leaving the practical deployment benefits insufficiently established.
  3. State-tracking accuracy degrades substantially under longer evaluation sequences, while the associative memory exhibits limited performance on repeated-key rebinding tasks.
Questions for authors
  1. How does the event-boundary execution schedule perform in end-to-end inference latency against optimized baselines when using learned gates and realistic batch sizes?
  2. Can the authors validate the claimed energy savings using direct hardware measurements that include memory access, control overhead, and other unaccounted operations?
  3. What modifications could improve long-context state retention and repeated-key rebinding without increasing the fixed memory budget or computational cost?
  4. Why do the authors not compare EGAM with Gated Slot Attention?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Zt9f (1/3), W1 seeds and scale, Q1 end-to-end latency with learned gates

Thank you, Reviewer Zt9f, for the constructive review and the four specific questions. We measured each one and report the results below. This comment covers W1 and Q1, the second covers Q2 and the third covers Q3 and Q4. Everything is in the revised PDF, in the new Table 5 (the controls table, which moves the FineWeb table to Table 6) and in Appendices A, C.5, D.3 and E.4. In that PDF, new content is blue and wording edits to the submitted text are green.

W1

The language modeling experiments use relatively small models and limited training budgets, with only one seed reported for the FineWeb-Edu evaluation.

Thank you. We agree that one seed is not enough. We trained two more FineWeb-Edu seeds for all four models at the FineWeb-Edu recipes. The FP perplexities for seeds 1234 / 1235 / 1236 are 29.328 / 29.367 / 29.370 for the combined model, 30.431 / 30.565 / 30.560 for Gated DeltaNet-2, 30.698 / 30.854 / 30.807 for Mamba2 and 31.213 / 31.241 / 31.188 for attention. The ordering of the four models is the same in every seed, with the combined model 1.1 to 1.2 PPL below the next model, and the paired two-term change is at most 0.040 PPL across the twelve runs. Table 6 now reports three-seed means, and the per-seed values are in Appendix C.5. On scale, 50.7M parameters at 236M tokens and 122.5M at 1.0B tokens are the budgets at which every baseline could be trained on one GPU with identical data order. The two-term change of the combined model stays small at both, 0.004 PPL at the smaller scale (Table 4) and 0.011 PPL at the larger one (Table 6).

Q1 / W2

How does the event-boundary execution schedule perform in end-to-end inference latency against optimized baselines when using learned gates and realistic batch sizes?

Thank you for asking for the full-model number. We measured the end-to-end latency of every Table 4 model and every FineWeb-Edu model with its learned gates. The runs use the same checkpoints and settings as those tables, embedding through vocabulary head, in bf16 on the GPU of Table 1. FLINT runs in the fused state path of Table 1, EGAM runs as compiled PyTorch, the baselines run in their library kernels, and no CUDA graphs are used. The tables give the median ms per forward for each B×LB \times L (batch size x sequence length). The xLSTM adaptation is configured for context 1024 and does not run at 4096.

WikiText-103 (50.7M), ms per forward at B x L1 x 102416 x 10241 x 409616 x 4096
Diagonal reference15.835.257.6152.0
Diagonal + FLINT (ours)22.140.968.4171.1
Diagonal + FLINT + EGAM (ours)25.745.372.7212.5
DeltaNet21.538.062.2169.6
Gated DeltaNet-222.340.366.0183.1
Gated DeltaProduct (k=2)22.038.163.5173.2
Mamba220.136.860.9161.9
xLSTM adaptation26.483.9--
Attention17.338.960.3175.1
FineWeb-Edu (122.5M), ms per forward at B x L1 x 102416 x 10241 x 409616 x 4096
Diagonal + FLINT + EGAM (ours)37.084.5107.6401.6
Gated DeltaNet-237.879.3103.3366.9
Mamba231.169.895.1311.8
Attention26.574.495.4344.8

The complete model takes 1.2 to 1.3x the time of DeltaNet and Mamba2 on WikiText-103. On FineWeb-Edu it takes 1.0 to 1.1x the time of Gated DeltaNet-2 and 1.1 to 1.3x that of Mamba2. Without EGAM, Diagonal + FLINT takes 1.01 to 1.10x the time of DeltaNet and 0.93 to 1.04x that of Gated DeltaNet-2.

Appendix E.3 isolates the schedule by running one trained recurrence, with weights, inputs and gates frozen, token by token and by event boundaries. The recurrence speedup of the event schedule grows with length and reaches 1.84x at L = 4096 with batch 16 and 2.73x with batch 1 (Table 15). The recurrence is one part of the forward pass. The MLP costs about 2.1M MACs per token and layer against 0.865M for the mixer (Appendix C.4). The savings of two-term weights are measured in Q2, on the hardware they target. The grid is in Appendix E.4, with one sentence in Sec. 5.1.

Q2 follows in the next comment.

AuthorsOpenReview ↗

Response to Reviewer Zt9f (2/3), Q2 direct hardware measurements

Q2 / W2

Can the authors validate the claimed energy savings using direct hardware measurements that include memory access, control overhead, and other unaccounted operations?

We agree that the analytic model alone does not establish deployment benefits, and we kept EarithE_{\mathrm{arith}} as the labeled arithmetic subtotal (Appendix A). We added three direct measurements.

(i) The hardware that the two-term projections target, an FPGA-prototyped RV32 core at 100 MHz with a multiplier-free datapath. Rail power comes from the board's PMBus monitors, and every checksum is bit-exact against a NumPy model. The module and mixer rows count every cycle on the core, so memory access and control are included.

On the RV32 coreint8 MACtwo-term shift-addtwo-term / int8
EGAM key projection, cycles per weight99.538.00.38x
EGAM key projection, dynamic energy per weight170.8 nJ62.2 nJ0.36x
EGAM module, cycles per token (equal dynamic power)40.9M16.6M0.41x
Combined mixer (FLINT + EGAM), cycles per token68.6M44.3M0.65x
Attention mixer, cycles per token at L = 256 / 1024 / 2048133.8M / 216.6M / 327.0Mtwo-term combined mixer at 0.33x / 0.20x / 0.14x of attention

(ii) GPU energy per forward on the GPU of Q1, where two-term weights run through the same floating-point units as FP weights. We read the NVML total-energy counter around synchronized groups of forward calls, at least 2 s per group and the median of five groups, with idle power measured on the free device and subtracted. The tables give mJ per forward, total with the idle-subtracted value in parentheses.

WikiText-103 (50.7M), mJ per forward at B x L, total (idle subtracted)1 x 102416 x 10241 x 409616 x 4096
Diagonal reference2,636 (1,229)17,512 (14,313)10,260 (5,010)76,439 (62,627)
Diagonal + FLINT (ours)3,599 (1,536)18,629 (14,880)12,246 (5,638)80,822 (65,299)
Diagonal + FLINT + EGAM (ours)3,943 (1,890)20,706 (17,000)13,719 (7,600)104,910 (87,590)
DeltaNet3,634 (1,513)18,891 (15,174)11,745 (5,574)82,776 (66,423)
Gated DeltaNet-23,787 (1,664)20,176 (16,325)12,027 (5,963)92,165 (74,658)
Gated DeltaProduct (k=2)3,670 (1,485)19,191 (15,497)12,107 (5,850)85,362 (68,639)
Mamba23,414 (1,497)18,379 (14,887)11,274 (5,373)79,390 (64,002)
xLSTM adaptation5,092 (2,501)40,273 (31,934)--
Attention3,181 (1,421)19,327 (15,392)11,515 (5,446)88,356 (70,659)
FineWeb-Edu (122.5M), mJ per forward at B x L, total (idle subtracted)1 x 102416 x 10241 x 409616 x 4096
Diagonal + FLINT + EGAM (ours)6,521 (3,074)43,792 (35,985)22,827 (12,218)204,502 (167,663)
Gated DeltaNet-26,471 (3,137)42,434 (35,388)21,525 (12,400)189,946 (157,583)
Mamba25,576 (2,658)37,586 (31,112)18,815 (9,869)164,806 (135,908)
Attention5,606 (2,995)39,429 (32,401)19,745 (10,945)186,165 (153,512)

On WikiText-103 the combined model uses 1.09 to 1.32x the total energy per forward of DeltaNet and Mamba2, and its perplexity is 4.0 lower than DeltaNet's. On FineWeb-Edu it uses 1.01 to 1.08x the total energy of Gated DeltaNet-2 and 1.17 to 1.24x that of Mamba2, at 1.2 and 1.4 lower perplexity.

(iii) A Jetson Orin (JetPack 6.2). The CPU+GPU rail energy comes from the board's INA3221 monitors with idle subtracted, and the tables give the median ms and the idle-subtracted mJ per forward.

WikiText-103 (50.7M), ms / mJ per forward at B x L1 x 10244 x 10241 x 4096
Diagonal reference241 / 759425 / 2,581934 / 3,205
Diagonal + FLINT + EGAM (ours)295 / 961425 / 2,9301,067 / 3,809
DeltaNet276 / 903391 / 2,866974 / 3,386
Gated DeltaNet-2276 / 941385 / 2,950941 / 3,452
Gated DeltaProduct (k=2)265 / 883362 / 2,735921 / 3,165
Mamba2260 / 864370 / 2,758899 / 3,232
xLSTM adaptation357 / 1,228707 / 4,954- / -
Attention253 / 862411 / 2,877987 / 3,558
FineWeb-Edu (122.5M), ms / mJ per forward at B x L1 x 10244 x 10241 x 4096
Diagonal + FLINT + EGAM (ours)446 / 1,802754 / 5,8551,639 / 7,145
Gated DeltaNet-2425 / 1,913710 / 6,0951,545 / 6,941
Mamba2392 / 1,753704 / 5,7591,449 / 6,589
Attention401 / 1,799767 / 6,2281,558 / 7,457

On the edge device the combined model uses 1.02 to 1.18x the rail energy of DeltaNet and Mamba2 on WikiText-103. On FineWeb-Edu it uses 1.02 to 1.08x that of Mamba2, 0.94 to 1.03x that of Gated DeltaNet-2 and 0.94 to 1.00x that of attention. The core measurements are in Appendix A, and the GPU energy and Orin tables are in Appendix E.4.

Q3 and Q4 follow in the next comment.

AuthorsOpenReview ↗

Response to Reviewer Zt9f (3/3), Q3 length and rebinding, Q4 Gated Slot Attention

Q3 / W3

What modifications could improve long-context state retention and repeated-key rebinding without increasing the fixed memory budget or computational cost?

Thank you for raising both. Each has a measured mitigation that keeps the fixed memory budget. Training at the target length leaves inference unchanged. Restoring the state from the store costs one write and one read per gap, and the erase-then-write rule costs one masked subtraction per admitted write.

Length

The degradation is at 4x the training length. Training at L = 512 gives 0.987 to 1.000 at that length (Appendix D.1), so training at the target length recovers the accuracy without a larger state. The joint store-once task, trained at L = 128, reaches 0.969 to 0.992 query accuracy at L = 512 with learned gates (Table 3). A second option uses the existing store across long gaps. Appendix D.4 writes the FLINT state into the associative store at a segment boundary and restores it after the gap, with a zero-overlap address code selected from 2,048 random candidates. Final-segment accuracy rises from 0.095 / 0.184 to 1.000 / 1.000 across one 4,096-token gap and from 0.013 / 0.006 to 0.860 / 0.893 across four (shadow-state / true-state gates). Both results match an exact copy of the state. This intervention uses oracle timing and non-leaky continuous storage.

In the language model, the combined WikiText-103 model reaches 26.78 / 26.82 / 26.64 PPL at L = 8192, 8x the training length, against 26.78 / 26.72 / 26.62 at L = 1024. The change is within 0.4% in every seed, while the attention model goes from 28.83 to 48.70 (new, Appendix E.4).

Rebinding

The additive store averages repeated writes to one key, which is the failure Table 3 isolates. The store update of Sec. 3.2 adds every admitted write into the selected slots,

Mt,m=λMt−1,m+gtwAt,mwvt,M_{t,m} = \lambda M_{t-1,m} + g^w_t A^w_{t,m} v_t,

so after STORE$_k$ with value v1v_1 and a later STORE$_k$ with v2v_2 the slots of key kk hold v1+v2v_1 + v_2 and QUERY$_k$ reads their mean. The erase-then-write rule replaces the content of the selected slots instead,

Mt,m=Mt−1,m+gtwAt,mw(vt−Mt−1,m),M_{t,m} = M_{t-1,m} + g^w_t A^w_{t,m} (v_t - M_{t-1,m}),

so the selected slots hold vtv_t after the write. The write counter Ct,mC_{t,m} becomes an occupancy flag, Ot,m=max⁡(Ot−1,m,gtwAt,mw)O_{t,m} = \max(O_{t-1,m}, g^w_t A^w_{t,m}), and the read averages the matched occupied slots. Addressing uses a fixed disjoint codebook. The 256 slots are split into four blocks of 64, STORE$_k$ writes to block kk and QUERY$_k$ reads block kk, so repeated writes to one key land on the same slots and never touch another key's slots. The state is unchanged (256 slots, no extra accumulator), and the write costs one masked subtraction more than the additive write. The table (new, Appendix D.3) gives rebinding query accuracy on the Table 3 protocol with oracle gates and this codebook, four cells per row (two learning rates x two seeds). Both rules use the codebook, so the additive row differs from the rebinding row of Table 3 (0.835 to 0.905 at L = 128), which uses the addressing of Sec. 3.2.

Rebinding query accuracyL = 128L = 512
Additive write0.809 to 0.8350.285 to 0.341
Erase-then-write0.999 to 1.0000.674 to 0.788

With the erase-then-write rule, rebinding queries are answered at 0.999 to 1.000 at the training length. At 4x that length the accuracy is at least 0.674, against at most 0.341 for the additive write. The language-model store keeps the additive write.

Q4

Why do the authors not compare EGAM with Gated Slot Attention?

Thank you for the pointer. It is the right baseline for a fixed-slot memory, and we added it with the same backbone, the same parameter matching and the Mamba2/xLSTM selection procedure (fla implementation, 4 heads x 128 slots). We selected the learning rate on seed 1234 from 3e-4, 6e-4 and 1.2e-3, which gave 31.607, 28.622 and 27.973 PPL, then trained seeds 1235 and 1236 at 1.2e-3 (28.133 and 27.873). The table gives the three-seed means.

Model (WikiText-103, 3 seeds)PPL FPPPL two-termMixer MACs (FP / two-term)State
Gated Slot Attention (4 heads x 128 slots)27.99328.0171.90M / 1.90M131,072
Diagonal + FLINT + EGAM26.70526.7090.865M / 0.469M131,584
Diagonal + EGAM, parameter-matched (no FLINT)26.48726.4960.601M / 0.204M131,456

At nearly the same state size the combined block is 1.3 PPL lower than GSA with 0.865M FP mixer MACs against 1.90M (0.469M with two-term weights). The memory-only control, Diagonal + EGAM, is 1.5 PPL lower than GSA with 0.601M. On the zero-shot extraction probes of Table 4, GSA scores 2.01 on SWDE and 3.07 on SQuAD against 2.01 and 5.47 for the combined model (FP, three seeds). The GSA row is in the new Table 5 and Sec. 5.4.

We are grateful for the constructive questions and hope that the measurements above address them.

Reviewer_xfihOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a recurrent block with two components. FLINT performs cheap updates at every token and more expressive updates at selected events. EGAM stores and retrieves associations using a fixed number of memory slots. The authors also approximate some projection weights so that multiplications can be replaced with additions and bit operations. Experiments cover state tracking, associative recall, execution speed, and language modeling.

Critical strengths
  • The separation between tracking and recall is well motivated. The joint synthetic task also provides a useful test of the same
  • The language-model comparisons are thorough and use matched parameter counts and training data, with three seeds on WikiText. The combined model reaches perplexity close to Mamba2 and xLSTM, and approximating the memory projections has little effect on perplexity across both datasets.
Critical weaknesses
  • The paper is quite hard to read. It extensively uses semicolons and reads heavily AI written/formatted. Phrases such as “event-proportional transition computation” make the explanation more abstract than necessary. More direct wording would help.
  • The practical efficiency benefit remains only partly validated. Table 1 shows blocks with event masks supplied in advance, while the arithmetic savings assume an execution path that is not timed on the GPU. The energy estimates also exclude memory movement, control, and several other costs. Overall, the results show lower arithmetic costs, but stronger inference measurements are needed to establish the practical advantage.
  • The language-model experiments do not isolate the value of learned event selection. Table 4 shows that adding EGAM to FLINT substantially improves perplexity, but does not include a matched model with EGAM and diagonal updates alone, or random or regularly spaced events at the same budget.
Questions for authors

Please see the weaknesses above

6 comments
AuthorsOpenReview ↗

Response to Reviewer xfih (1/3), event-selection controls (W3)

We are grateful to Reviewer xfih for the thorough review. For W3 we trained the controls you listed. For W2 we measured end-to-end latency with learned gates, and we measured the shift-add path on hardware where it applies. For W1 we rewrote the densest sentences in plain words and removed the semicolons. This comment covers W3, and the next two cover W1 and W2. The new WikiText-103 runs use the Table 4 recipe with three seeds, and everything is in the revised PDF (the new controls table is Table 5, so the FineWeb table becomes Table 6). In that PDF, new content is blue and wording edits to the submitted text are green.

W3

The language-model experiments do not isolate the value of learned event selection. Table 4 shows that adding EGAM to FLINT substantially improves perplexity, but does not include a matched model with EGAM and diagonal updates alone, or random or regularly spaced events at the same budget.

Thank you for naming the exact controls. We agree they were missing, and we trained all of them. They isolate where learned event selection matters. On sparse S5 the learned gate tracks the permutation at 0.9997 to 1.000, and a random schedule at the same event rate reaches 0.26 to 0.27 (S5 table below). On WikiText-103 perplexity the learned, random and periodic schedules are within seed variation of each other. The table gives perplexity in FP and with two-term weights. WikiText-103 rows are three-seed means at the Table 4 recipe, and the FineWeb-Edu rows are three-seed means at the FineWeb-Edu recipe.

ModelParamsPPL FPPPL two-term
WikiText-103, combined, learned gate (Table 4)50.7M26.70526.709
WikiText-103, Diagonal + EGAM, combined width (FLINT removed, same init)48.6M26.77126.786
WikiText-103, Diagonal + EGAM, parameter-matched (MLP 2275)50.7M26.48726.496
WikiText-103, combined, Bernoulli(0.10) events50.7M26.71726.725
WikiText-103, combined, every 10th token an event50.7M26.65326.659
FineWeb-Edu, combined, learned gate (FineWeb-Edu table)122.5M29.35529.366
FineWeb-Edu, Diagonal + EGAM, combined width (FLINT removed, same init)115.4M29.61629.612

Removing FLINT at the same width costs +0.05 to +0.09 PPL per seed on WikiText-103 and +0.23 to +0.28 per seed on FineWeb-Edu. Moving its parameters into the MLP gains 0.22 PPL. A random or periodic schedule is within 0.13 PPL of the learned gate in every seed (seed SD 0.08).

In the revision, Table 5 and a paragraph in Sec. 5.4 carry these controls, and Table 6 carries the FineWeb control row.

We also looked at where FLINT matters in the same WikiText evaluation. We split the 40 validation batches (655k tokens per seed) by quotation marks and compared the combined model with the two models without FLINT. The table gives perplexity for seeds 1234 / 1235 / 1236.

TokensWith FLINTWithout FLINT, same widthWithout FLINT, parameter-matched
Inside quotations (8.4%)32.748 / 32.934 / 32.66433.149 / 33.143 / 33.16532.916 / 32.803 / 32.730
Outside quotations25.946 / 25.856 / 25.78525.960 / 25.906 / 25.83125.650 / 25.619 / 25.605

Against the same-width model, the gain is larger inside quoted speech in every seed. The parameter-matched control is 0.18 to 0.30 PPL lower than the combined model outside quotations. Inside quotations the two are within 0.17 PPL.

In two of the three seeds one FLINT layer fires at nearly every quotation mark, and turning the FLINT events off at inference raises the perplexity inside quotations by 1.0 and 2.5 PPL against 0.3 and 0.2 outside.

SeedEvent rate at quote tokens (that layer's overall rate)PPL inside quotations, events on / offPPL outside, events on / off
12350.98 (0.56)32.93 / 33.9625.86 / 26.18
12361.00 (0.04)32.66 / 35.1625.79 / 26.01

The sparse S5 task of Table 2 separates the learned gate from a random schedule. The table trains a block with the Table 4 organization on that task at the Table 2 width and L = 128, with only the gate changed between rows and the FLINT branch removed in the last row. Each cell is the range over four runs of each run's best per-position accuracy.

Table 4 block organization on sparse S5L = 128L = 512
With FLINT, learned shadow-state gate (the LM's gate)0.9997 to 1.0000.83 to 0.91
With FLINT, Bernoulli(0.12) schedule0.26 to 0.270.08
FLINT branch removed0.18 to 0.190.05

Without FLINT, the block does not track the permutation (0.18 to 0.19), and neither does a random schedule at the same event rate (0.26 to 0.27). The learned gate does, and it reaches 0.83 to 0.91 at four times the training length.

The quotation analysis and this table are in Appendix E.5.

W1 and W2 follow in the next two comments.

AuthorsOpenReview ↗

Response to Reviewer xfih (2/3), readability (W1) and practical efficiency (W2), learned-gate latency and the shift-add path

W1

The paper is quite hard to read. It extensively uses semicolons [...] Phrases such as "event-proportional transition computation" make the explanation more abstract than necessary. More direct wording would help.

Thank you for pointing this out. We agree that the wording was more abstract than necessary. We removed the semicolons from the text and the captions and rewrote the densest sentences in plain words. The phrase you quote was the title of our first contribution. The title is now "Transition work that scales with the number of events", and the item states that this work is done only at the EE fired events. The wording edits are green in the revised PDF.

W2

The practical efficiency benefit remains only partly validated. Table 1 shows blocks with event masks supplied in advance, while the arithmetic savings assume an execution path that is not timed on the GPU. The energy estimates also exclude memory movement, control, and several other costs. Overall, the results show lower arithmetic costs, but stronger inference measurements are needed to establish the practical advantage.

Thank you for separating the three points. We agree with each of them and answer them in turn, (a) and (b) here and (c) in the next comment.

(a) Known masks in Table 1

Table 1 isolates the schedule by design. We added the end-to-end latency of every Table 4 model and every FineWeb-Edu model with its learned gates, using the same checkpoints and settings, embedding through vocabulary head, bf16, and the same GPU as Table 1. FLINT runs in the fused state path of Table 1, EGAM runs as compiled PyTorch, the baselines run in their library kernels, and no CUDA graphs are used. The tables give the median ms per forward for each B×LB \times L (batch size x sequence length). The xLSTM adaptation is configured for context 1024 and does not run at 4096.

WikiText-103 (50.7M), ms per forward at B x L1 x 102416 x 10241 x 409616 x 4096
Diagonal reference15.835.257.6152.0
Diagonal + FLINT (ours)22.140.968.4171.1
Diagonal + FLINT + EGAM (ours)25.745.372.7212.5
DeltaNet21.538.062.2169.6
Gated DeltaNet-222.340.366.0183.1
Gated DeltaProduct (k=2)22.038.163.5173.2
Mamba220.136.860.9161.9
xLSTM adaptation26.483.9--
Attention17.338.960.3175.1
FineWeb-Edu (122.5M), ms per forward at B x L1 x 102416 x 10241 x 409616 x 4096
Diagonal + FLINT + EGAM (ours)37.084.5107.6401.6
Gated DeltaNet-237.879.3103.3366.9
Mamba231.169.895.1311.8
Attention26.574.495.4344.8

With learned gates the complete model takes 1.2 to 1.3x the time of DeltaNet and Mamba2 on WikiText-103. On FineWeb-Edu it takes 1.1 to 1.3x the time of Mamba2 and 1.0 to 1.1x that of Gated DeltaNet-2. The full grid is in Appendix E.4, with one sentence in Sec. 5.1.

(b) The shift-add path

You are right that the GPU does not time this path. Our GPU implementation runs two-term weights through the same floating-point units as FP weights. We measured the path on the hardware it targets, an FPGA-prototyped RV32 core at 100 MHz with a multiplier-free datapath. Rail power comes from the board's PMBus monitors, and every checksum is bit-exact against a NumPy model.

On the RV32 coreint8 MACtwo-term shift-addtwo-term / int8
EGAM key projection, cycles per weight99.538.00.38x
EGAM key projection, dynamic energy per weight170.8 nJ62.2 nJ0.36x
EGAM module, cycles per token (equal dynamic power)40.9M16.6M0.41x
Combined mixer (FLINT + EGAM), cycles per token68.6M44.3M0.65x
Attention mixer, cycles per token at L = 256 / 1024 / 2048133.8M / 216.6M / 327.0Mtwo-term combined mixer at 0.33x / 0.20x / 0.14x of attention

These are three new paragraphs in Appendix A.

AuthorsOpenReview ↗

Response to Reviewer xfih (3/3), practical efficiency (W2), energy measured on the GPU and on an edge SoC

(c) Energy exclusions

We agree. EarithE_{\mathrm{arith}} is the mixer arithmetic subtotal, and Appendix A lists what it leaves out. We kept that scope and added measurements that include memory movement and control. The core numbers in the previous comment count every cycle of the complete module and mixer on the core, so both are included there. On the GPU of Table 1 we read the NVML total-energy counter around synchronized groups of forward calls (at least 2 s per group, median of five, idle power measured on the free device and subtracted). The tables give mJ per forward for every Table 4 model and every FineWeb-Edu model, total with the idle-subtracted value in parentheses.

WikiText-103 (50.7M), mJ per forward at B x L, total (idle subtracted)1 x 102416 x 10241 x 409616 x 4096
Diagonal reference2,636 (1,229)17,512 (14,313)10,260 (5,010)76,439 (62,627)
Diagonal + FLINT (ours)3,599 (1,536)18,629 (14,880)12,246 (5,638)80,822 (65,299)
Diagonal + FLINT + EGAM (ours)3,943 (1,890)20,706 (17,000)13,719 (7,600)104,910 (87,590)
DeltaNet3,634 (1,513)18,891 (15,174)11,745 (5,574)82,776 (66,423)
Gated DeltaNet-23,787 (1,664)20,176 (16,325)12,027 (5,963)92,165 (74,658)
Gated DeltaProduct (k=2)3,670 (1,485)19,191 (15,497)12,107 (5,850)85,362 (68,639)
Mamba23,414 (1,497)18,379 (14,887)11,274 (5,373)79,390 (64,002)
xLSTM adaptation5,092 (2,501)40,273 (31,934)--
Attention3,181 (1,421)19,327 (15,392)11,515 (5,446)88,356 (70,659)
FineWeb-Edu (122.5M), mJ per forward at B x L, total (idle subtracted)1 x 102416 x 10241 x 409616 x 4096
Diagonal + FLINT + EGAM (ours)6,521 (3,074)43,792 (35,985)22,827 (12,218)204,502 (167,663)
Gated DeltaNet-26,471 (3,137)42,434 (35,388)21,525 (12,400)189,946 (157,583)
Mamba25,576 (2,658)37,586 (31,112)18,815 (9,869)164,806 (135,908)
Attention5,606 (2,995)39,429 (32,401)19,745 (10,945)186,165 (153,512)

On WikiText-103 the combined model uses 1.09 to 1.32x the total energy per forward of DeltaNet and Mamba2, and its perplexity is 4.0 lower than DeltaNet's. On FineWeb-Edu it uses 1.01 to 1.08x the total energy of Gated DeltaNet-2 and 1.17 to 1.24x that of Mamba2, at 1.2 and 1.4 lower perplexity. We repeated the measurement on a Jetson Orin (JetPack 6.2), reading the CPU+GPU rail with the board's INA3221 monitors and subtracting idle. The tables give the median ms and the idle-subtracted mJ per forward.

WikiText-103 (50.7M), ms / mJ per forward at B x L1 x 10244 x 10241 x 4096
Diagonal reference241 / 759425 / 2,581934 / 3,205
Diagonal + FLINT + EGAM (ours)295 / 961425 / 2,9301,067 / 3,809
DeltaNet276 / 903391 / 2,866974 / 3,386
Gated DeltaNet-2276 / 941385 / 2,950941 / 3,452
Gated DeltaProduct (k=2)265 / 883362 / 2,735921 / 3,165
Mamba2260 / 864370 / 2,758899 / 3,232
xLSTM adaptation357 / 1,228707 / 4,954- / -
Attention253 / 862411 / 2,877987 / 3,558
FineWeb-Edu (122.5M), ms / mJ per forward at B x L1 x 10244 x 10241 x 4096
Diagonal + FLINT + EGAM (ours)446 / 1,802754 / 5,8551,639 / 7,145
Gated DeltaNet-2425 / 1,913710 / 6,0951,545 / 6,941
Mamba2392 / 1,753704 / 5,7591,449 / 6,589
Attention401 / 1,799767 / 6,2281,558 / 7,457

On the edge device the combined model uses 1.02 to 1.18x the rail energy of DeltaNet and Mamba2 on WikiText-103. On FineWeb-Edu it uses 1.02 to 1.08x that of Mamba2, 0.94 to 1.03x that of Gated DeltaNet-2 and 0.94 to 1.00x that of attention. Both measurements are new tables in Appendix E.4.

We appreciate the care in your review and hope that these measurements and the revised text address the concerns you raised.

Reviewer_R6UNOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a linear sequence-modeling block that separates transition computation from associative storage into two complementary modules: FLINT and EGAM. FLINT controls transition work, EGAM controls associative storage.

Critical strengths
  1. Clear resource decoupling with length-independent state accounting. FLINT controls event-conditioned transition work, EGAM controls associative storage, and APoT controls arithmetic representation. Task-state size is independent of L.
  2. Controlled synthetic validation isolates the core mechanisms. The S5 tracking, contextual-gate, store-once, and rebinding experiments separately test event selection, state tracking, and associative interference, with gate batteries and transition counts. This matters because it provides mechanistic evidence that event-conditioned tracking and bounded recall can complement each other.
Critical weaknesses
  1. Language-model gains come mainly from EGAM, not FLINT. The core FLINT contribution is not supported at the language-model level. Diagonal+FLINT reaches PPL 36.736, clearly worse than DeltaNet 30.726 and Gated DeltaNet-2 29.325; the combined model only reaches 26.705 after adding EGAM, and supervised SQuAD also favors DeltaNet/xLSTM. This is consequential because the paper’s central claim is that event-conditioned transitions are a useful modeling primitive, yet its LM gains appear to come mainly from associative memory, leaving FLINT validated mainly on synthetic tasks.
Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer R6UN (1/2), perplexity and the quotation analysis

We sincerely thank Reviewer R6UN for the careful review. The weakness you raise concerns what FLINT contributes in the language model. We measured that directly, and the results are below. Everything is in the revised PDF (the new controls table is Table 5, so the FineWeb-Edu table becomes Table 6). In that PDF, new content is blue and wording edits to the submitted text are green.

W1

Language-model gains come mainly from EGAM, not FLINT. The core FLINT contribution is not supported at the language-model level. Diagonal+FLINT reaches PPL 36.736, clearly worse than DeltaNet 30.726 and Gated DeltaNet-2 29.325; the combined model only reaches 26.705 after adding EGAM, and supervised SQuAD also favors DeltaNet/xLSTM. [...] leaving FLINT validated mainly on synthetic tasks.

We agree with this reading. The perplexity of the full block comes from EGAM, as Table 4 shows (Diagonal + FLINT 36.7, with EGAM 26.7), and we do not claim otherwise. Our claim for FLINT is state tracking. In the language models, removing FLINT at the same width raises perplexity in every seed on both corpora (part 1), and on WikiText-103 the gain is larger inside quoted speech (part 2). Part 3 measures tracking on sparse S5S_5 with the Table 4 block organization.

1. Perplexity with and without the FLINT branch

The first control has the same width and initialization, with only the FLINT branch removed. The second also widens the MLP so that the parameter count matches the combined model. WikiText-103 and FineWeb-Edu are means of three seeds (Table 4 and FineWeb-Edu recipes).

DatasetModelParamsPPL FPPPL two-term
WikiText-103Diagonal + EGAM, same width (no FLINT)48.6M26.77126.786
WikiText-103Diagonal + EGAM, parameter-matched (no FLINT)50.7M26.48726.496
WikiText-103Diagonal + FLINT + EGAM50.7M26.70526.709
FineWeb-EduDiagonal + EGAM, same width (no FLINT)115.4M29.61629.612
FineWeb-EduDiagonal + FLINT + EGAM122.5M29.35529.366

Without FLINT at the same width, perplexity is higher by +0.048, +0.058 and +0.090 per seed on WikiText-103 and by +0.283, +0.230 and +0.271 per seed on FineWeb-Edu. The parameter-matched control is 0.22 lower than the combined model on WikiText-103 (26.487 against 26.705). FLINT's contribution is state tracking, which parts 2 and 3 measure. The rows are in the new Table 5 and Table 6.

2. What FLINT does in the trained LM

We split the 40 validation batches of the Table 4 evaluation (655k tokens per seed) by quotation marks. The table gives WikiText-103 perplexity for seeds 1234 / 1235 / 1236.

TokensWith FLINTWithout FLINT, same widthWithout FLINT, parameter-matched
Inside quotations (8.4%)32.748 / 32.934 / 32.66433.149 / 33.143 / 33.16532.916 / 32.803 / 32.730
Outside quotations25.946 / 25.856 / 25.78525.960 / 25.906 / 25.83125.650 / 25.619 / 25.605

Against the same-width model, FLINT's gain is larger inside quoted speech, 0.2 to 0.5 PPL in every seed against 0.01 to 0.05 outside. The parameter-matched control is 0.18 to 0.30 PPL lower than the combined model outside quotations. Inside quotations the two are within 0.17 PPL. In two of the three seeds one FLINT layer fires at nearly every quotation mark, and turning the FLINT events off at inference raises the perplexity inside quotations by 1.0 and 2.5 PPL against 0.3 and 0.2 outside.

SeedEvent rate at quote tokens (that layer's overall rate)PPL inside quotations, events on / offPPL outside, events on / off
12350.98 (0.56)32.93 / 33.9625.86 / 26.18
12361.00 (0.04)32.66 / 35.1625.79 / 26.01

In the WikiText models, the FLINT branch therefore contributes to prediction inside quoted speech. In seeds 1235 and 1236 it depends on the FLINT events, and one layer places them at quotation marks. This is consistent with a state that marks quoted spans. The quotation analysis is the new Appendix E.5, with one sentence in Sec. 5.4.

The tracking evaluation and the two comparisons follow in the next comment.

AuthorsOpenReview ↗

Response to Reviewer R6UN (2/2), sparse S5 tracking and the two comparisons

3. Tracking with the Table 4 block organization

We use S5 permutation tracking because diagonal state-space models cannot solve it under a standard complexity assumption (Merrill et al., 2024). Grazzi et al. (2025) and DeltaProduct (Siems et al., 2025) use it to measure what non-diagonal updates add. The table below trains a block with the Table 4 organization (PDiag, FLINT, EGAM, MLP) on this task at the Table 2 width and L = 128, with only the gate changed between rows and the FLINT branch removed in the last row. Each cell is the range over four runs of each run's best per-position accuracy.

Table 4 block organization on sparse S5L = 128L = 512
With FLINT, learned shadow-state gate (the LM's gate)0.9997 to 1.0000.83 to 0.91
With FLINT, learned LIF gate (the Table 2 gate)0.999 to 1.0000.77 to 0.82
With FLINT, Bernoulli(0.12) schedule0.26 to 0.270.08
FLINT branch removed0.18 to 0.190.05

Without FLINT, the block does not track the permutation (0.18 to 0.19), and neither does a random schedule at the same event rate (0.26 to 0.27). Both learned gates track it, and the one the language model uses has the higher accuracy at four times the training length (0.83 to 0.91).

4. The two comparisons you mention

  • Diagonal + FLINT (36.7) is the tracking-only reference. Its state is a 256-element vector at 0.461M mixer MACs per token and layer, against a 65,664-element matrix state at 1.445M for DeltaNet. FLINT with a learned gate tracks sparse S5S_5 (Table 2), which DeltaNet, Gated DeltaNet and Gated DeltaNet-2 do not learn in any of their 48 cells of our protocol (Appendix C.3). It shows what FLINT adds to the diagonal baseline and what EGAM adds on top. We agree that the row reads as a direct competitor in Table 4, so we have added a sentence to the caption that labels it as an ablation step.
  • About SQuAD, you are right that DeltaNet and xLSTM score higher, and we report that result unchanged (Appendix E.2). On supervised SQuAD they reach 27.54 and 27.57 F1 against 21.61 for the combined model, 22.75 for Mamba2 and 15.15 for attention (three-seed means). The paper's claim is quality retention under a fixed state and arithmetic budget, together with the tracking above. The combined model reaches 26.7 PPL at one third of DeltaNet's mixer MACs and with 51% less estimated mixer arithmetic energy.

We appreciate your engagement with our work and hope that these additional experiments address your remaining concerns.

References

  • Merrill, Petty, Sabharwal. The Illusion of State in State-Space Models. ICML 2024.
  • Grazzi, Siems, Zela, Franke, Hutter, Pontil. Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues. ICLR 2025.
  • Siems, Carstensen, Zela, Hutter, Pontil, Grazzi. DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products. NeurIPS 2025.
AuthorsOpenReview ↗

Gentle follow-up on our response

Dear Reviewer R6UN,

Thank you again for your time and thoughtful comments on our paper. We have carefully addressed the concerns and questions raised in your review and posted detailed responses, including additional analyses and experimental results, together with a revised PDF.

As the discussion period is approaching its end, we would like to kindly follow up in case you have not yet had a chance to read our responses. We hope they help clarify the concerns raised in your review, and we would be glad to provide any further clarification.

Thank you again for your time and consideration.

Best regards,
The Authors

AuthorsOpenReview ↗

Additional result for Reviewer R6UN, sparse S5 tracking with the Table 5 language models

Dear Reviewer R6UN,

Since our response, we have run the tracking test of part 3 with the trained language models of Table 5. Part 3 used a small block with the Table 4 organization. Here each WikiText-103 checkpoint of Table 5 is fine-tuned on the same sparse S5 task. The embedding and the tied output layer are replaced by ones for the 120 group elements, every other weight starts from the checkpoint, and every row uses one recipe (15k steps, AdamW at 1.2e-3). The table gives per-position accuracy at the training length L = 128 on 2,048 held-out sequences. Its last row is a reference without a model, a predictor that copies the last group token and never composes.

Table 5 checkpoint, fine-tuned on sparse S5seed 1234seed 1235seed 1236
Combined, learned events (ours)0.9940.9540.980
Combined, Bernoulli(0.10) events0.4210.5200.625
Combined, every-10th-token events0.6340.4470.679
Diagonal + EGAM, parameter-matched (MLP 2275)0.1590.3280.195
Diagonal + EGAM, combined width (FLINT removed)0.2330.2330.169
Reference without a model, copy the last group token0.1590.1590.159

The checkpoints without FLINT reach at most 0.328, and those with random or periodic events at most 0.679. The checkpoints with FLINT and its learned gate reach 0.954 to 0.994. The table is new in Appendix E.5 of the revised PDF, next to the small-block table of part 3.

We sincerely appreciate the time and effort you have devoted to reviewing our submission, and we would be glad to provide any further clarification.

Sincerely,
The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair and Reviewers,

We sincerely thank you for your time and constructive feedback. Below is a brief summary of how we addressed each comment. In the revised PDF, new content is blue and wording edits are green.

  • What FLINT contributes in the language model (R6UN). We agree that the perplexity of the combined model comes from EGAM, and our claim for FLINT is state tracking. We ran three new analyses of FLINT in the language models. First, we removed the FLINT branch at the same width and retrained. Without FLINT, perplexity is 0.05 to 0.09 PPL higher per seed on WikiText-103 and 0.23 to 0.28 higher on FineWeb-Edu (Tables 5 and 6). Second, we split the WikiText-103 validation tokens by quotation marks. Against that same-width model, the gain from FLINT is 0.2 to 0.5 PPL inside quotations and 0.01 to 0.05 outside in every seed (Appendix E.5). Third, we turned the FLINT events off at inference. In two of the three seeds one FLINT layer has an event rate of 0.98 and 1.00 at quotation marks, against that layer's overall rate of 0.56 and 0.04. In those seeds, turning the events off raises perplexity by 1.0 and 2.5 PPL inside quotations against 0.3 and 0.2 outside (Appendix E.5). This is consistent with a state that marks quoted spans. The Table 4 caption now labels the Diagonal + FLINT row as an ablation step.
  • Controls for FLINT and event selection (xfih, R6UN). We trained every control Reviewer xfih listed, at the Table 4 recipe with three seeds (Table 5). They are Diagonal + EGAM without FLINT, at the same width and at matched parameters, and the combined model with random and with periodic events. We also trained a block with the Table 4 organization on sparse S5S_5, at the Table 2 width and L = 128. These controls isolate where learned event selection matters. On sparse S5S_5 the block tracks the permutation at 0.9997 to 1.000 accuracy with its learned gate. It reaches 0.26 to 0.27 with a random schedule at the same event rate and 0.18 to 0.19 with the FLINT branch removed (Appendix E.5). On WikiText-103 perplexity the random and periodic schedules are within 0.13 PPL of the learned gate in every seed (seed SD 0.08).
  • Measured efficiency (xfih, Zt9f). We measured the shift-add path on the hardware it targets, an FPGA-prototyped RV32 core at 100 MHz with a multiplier-free datapath. On that core the EGAM module with two-term weights takes 0.41x the cycles of its int8 version, and that count includes memory access and control (Appendix A). The two-term combined mixer takes 0.33x, 0.20x and 0.14x the cycles of the int8 attention mixer at L = 256, 1024 and 2048. We also measured the end-to-end latency of every Table 4 and FineWeb-Edu model with its learned gates, and energy on the GPU and on a Jetson Orin (Appendix E.4). On the GPU, where two-term weights run through the same floating-point units as FP weights, the combined model takes 0.98 to 1.31x the time of DeltaNet, Gated DeltaNet-2 and Mamba2.
  • Seeds (Zt9f). We trained two more FineWeb-Edu seeds for all four models, and the ordering is the same in every seed (Table 6).
  • Length and rebinding (Zt9f). For length, we evaluated the combined WikiText-103 model at L = 8192, which is 8x the training length. Its perplexity is within 0.4% of its value at the training length in every seed (Appendix E.4). For rebinding, we added an erase-then-write rule that replaces the content of the selected slots. It keeps the same 256 slots and adds one masked subtraction per admitted write. On the Table 3 protocol with oracle gates and a fixed codebook, it answers rebinding queries at 0.999 to 1.000 accuracy at the training length. The additive write reaches 0.809 to 0.835 under the same conditions (Appendix D.3).
  • Gated Slot Attention (Zt9f). We added it with the same backbone and parameter matching. The combined model is 1.3 PPL lower, with a state of 131,584 elements against 131,072 (Table 5).
  • Readability (xfih). We removed the semicolons from the text and the captions and rewrote the densest sentences in plain words, including three sentences of the abstract.

Table 5 is new, so the FineWeb-Edu table is now Table 6, and the abstract now reports the three-seed FineWeb-Edu perplexity. Point-by-point responses are under each review.

We believe these additions and revisions address the concerns raised. We are deeply grateful to the Area Chair and the reviewers for the time and effort devoted to our submission.

Sincerely,
The Authors

AuthorsOpenReview ↗

Follow-up to the Rebuttal Summary

Dear Area Chair and Reviewers,

This comment adds the discussion that followed the summary above. In the second revised PDF, the new content is blue.

Reviewer xfih replied that "the revised presentation and hardware measurements address several of my key concerns" and named two remaining concerns. We ran a new measurement for each.

  • Parameter-matched control and learned event selection (xfih). This concern is about WikiText-103 perplexity (Table 5). There the parameter-matched Diagonal + EGAM model is 0.22 PPL lower than the combined model (26.487 against 26.705). The learned, random and periodic event schedules are within seed variation of each other (26.705, 26.717 and 26.653, seed SD 0.08). Our claim for FLINT is state tracking, which perplexity does not measure. To measure it on the same language models, we fine-tuned the combined model and the four controls of Table 5 on the sparse S5S_5 task of Table 2. Each model starts from its WikiText-103 checkpoint, and all five use one recipe and three seeds. At L = 128 the combined model with learned events reaches 0.954 to 0.994 accuracy. With random or periodic events at the event rate the learned gate is trained to (0.10), it reaches at most 0.679. The two models without FLINT reach at most 0.328 (Table 22, Appendix E.5). A predictor that copies the most recent group token and never composes scores 0.159.
  • End-to-end speed on the GPU and the Jetson (xfih). We agree that the lower arithmetic cost gives no end-to-end speedup on these devices, and we claim none. Both have hardware multipliers, so two-term weights save no time there. The combined model takes 0.98x to 1.31x the time of DeltaNet, Gated DeltaNet-2, Gated DeltaProduct and Mamba2 in every cell of the GPU and Jetson tables (Appendix E.4). The arithmetic saving applies to hardware without a multiplier. The summary above compares the combined mixer with the attention mixer on the FPGA-prototyped RV32 core. We have now run the mixers of the remaining Table 4 baselines on the same core. With int8 multiply-accumulates in every mixer, the combined mixer takes fewer cycles than every baseline mixer, at 0.21x to 0.52x. Two-term weights lower this to 0.14x to 0.34x (Appendix A). Within a session the dynamic power of the kernels is equal within 1%, so these are also the energy ratios on this core.

We also posted the S5S_5 table to Reviewer R6UN and the core table to Reviewer Zt9f, since they add to our earlier answers to those reviewers.

The second revised PDF adds Table 22 with its text in Appendix E.5, a table with its text in Appendix A, and one sentence each in Sections 5.1 and 5.4. The title, the abstract and the rest of the text are unchanged.

We believe these two measurements address the remaining concerns. We are deeply grateful to the Area Chair and the reviewers for their time and effort.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.