- Rating
- 4(+2)
- Confidence
- 4
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes Function-Valued Linear Attention (FVLA), an augmentation to matrix-state linear attention. The method expands the read/write address of a Delta-style recurrent memory from to by adding learned separable quadratic features . The resulting recurrent state has size , so and correspond to 2× and 5× main fast-state size respectively. The authors instantiate FVLA on Gated DeltaNet and Kimi Delta Attention and report improvements in language modeling and in-context retrieval at 140M and 340M scales. The paper also includes ablations on normalization, widened linear states, random response directions, readout calibration, response-mode scaling, and long-context S-NIAH extrapolation.
Critical strengths
-
Simple and modular construction.
FVLA is easy to understand as an expanded feature/state space for Delta-style linear attention. It preserves the host’s recurrent update structure and can in principle be attached to GDN/KDA-like models. -
Consistent gains over the chosen hosts.
At both 140M and 340M scale, $r=4 $ improves WikiText perplexity and retrieval averages over the corresponding GDN/KDA hosts. The paired training setup against GDN/KDA is a positive aspect. -
Useful ablations.
The paper includes controls for write normalization, widened linear state, random quadratic directions, learned readout calibration, and full quadratic features. These help support the claim that nonlinear feature/state expansion is beneficial. -
Maintains fixed recurrent memory w.r.t. sequence length.
FVLA increases the constant recurrent state size but does not introduce a KV cache or a memory that grows with context length.
Critical weaknesses
1. FVLA is a dense state/feature expansion, and the efficiency tradeoff is under-characterized.
FVLA expands the recurrent address from to , so and use 2× and 5× main recurrent matrix state. All expanded blocks are densely read/written at every token, so the activation rate is effectively 100%. The paper reports useful throughput numbers for and some operator-level latency scaling, but the strongest gains come from . I would like to see complete end-to-end efficiency: prefill/decode latency, training throughput, peak memory, and preferably state-I/O or kernel latency. The batch-size-1 inference timing is useful for latency but does not fully characterize throughput-saturated serving.
2. The MoM-GDN comparison is present, but not controlled for activated state capacity.
The issue is not that MoM is absent: the paper includes a MoM-GDN public-checkpoint reference. However, MoM expands memory through routed states with sparse activation, while FVLA activates all expanded state blocks densely. Thus the comparison does not isolate total state, active state per token, activation ratio, state I/O, or throughput. Please clarify the MoM configuration and compare under matched activated state size or matched throughput.
3. GDN2 may cover FVLA’s gain without doubling recurrent-state I/O.
FVLA doubles the recurrent matrix state and roughly doubles fast-state I/O per recurrent decoding step. A stronger update-rule method such as GDN2 may achieve similar gains over GDN/KDA without this 2× state expansion. I would like to see FVLA compared with GDN2 under the same training setup, reporting quality versus recurrent state size, decode-time state I/O, and throughput.
4. The polynomial feature alone is not isolated.
FVLA introduces nonlinear polynomial features together with expanded recurrent state capacity. I would like to see how much benefit the same polynomial feature gives to the GDN backbone at fixed state size, e.g. by replacing or compressing the address back to dimension . This would separate nonlinear feature-map gains from state-capacity gains.
5. Hybrid attention evaluation is missing.
All main experiments use pure GDN/KDA-style recurrent hosts. Since many competitive efficient LLMs use hybrid softmax/linear attention, and softmax layers may already improve retrieval, it is unclear whether FVLA remains beneficial in hybrid architectures.
6. Long-context evidence is encouraging but not broad enough.
The S-NIAH and streaming LM results are useful, but the long-context evaluation is still concentrated on synthetic needle retrieval. Gains also appear strongest on S-NIAH-1, while S-NIAH-2/3 remain more mixed.
7. The polynomial feature may provide gains without extra decoding I/O.
Related to point 4, if FVLA-style polynomial features improve GDN at fixed state size, then part of the reported gain may be achievable without doubling decoding state I/O. This ablation would be important for interpreting the efficiency benefit of FVLA.
8. Naive state expansion is only partially addressed.
The paper includes a small-scale 2× Wide GDN control, but the main results include , i.e. 5× recurrent state. I would like to see stronger naive expansion baselines at the main scale, such as larger key dimension or larger value dimension.
Questions for authors
Please respond to my concerns presented in the Weaknesses section. I would be happy to adjust my rating if these issues are well addressed.
11 comments
[Part2] Response to Reviewer e4Jw
Weakness 2:
The MoM-GDN comparison is present, but not controlled for activated state capacity.
Please clarify the MoM configuration and compare under matched activated state size or matched throughput.
Thank you for highlighting the distinction between sparse memory routing and dense response-space expansion. We have clarified the MoM-GDN configuration and added explicit measurements of total and active parameters, recurrent-state capacity, and training efficiency.
MoM-GDN uses four routed memories with top-2 selection and an additional shared memory. In contrast, FVLA expands the query-dependent response space within each recurrent memory, with all expanded state blocks actively updated.
| Model | Avg₆ ↑ | Total / Active Params. (M) | Resident / Active State (MiB) | Training (K tok/s) ↑ |
|---|---|---|---|---|
| GDN + FVLA (r=4) | 29.82 | 354.37 / 354.37 | 60 / 60 | 25.22 |
| MoM-GDN | 26.57 | 512.35 / 411.29 | 120 / 72 | 11.38 |
The results show that FVLA achieves higher aggregate retrieval accuracy with fewer total and active parameters, smaller recurrent-state capacity, and higher measured training throughput. The corresponding logical state read/write volumes are 120 MiB and 144 MiB per token for FVLA and MoM-GDN, respectively.
Importantly, MoM and FVLA expand linear-attention memory along different dimensions: MoM selects among multiple memory banks, whereas FVLA enriches the response functions represented within each bank. These mechanisms are therefore complementary and could, in principle, be combined by incorporating FVLA into MoM's routed memories.
We have added the capacity breakdown to Appendix D.1 and expanded Section 4.7 (Figure 6) and Appendix I (Efficiency Analysis) with the corresponding training and inference measurements.
[Part3] Response to Reviewer e4Jw
Weakness 3:
GDN2 may cover FVLA’s gain without doubling recurrent-state I/O.
FVLA doubles the recurrent matrix state and roughly doubles fast-state I/O per recurrent decoding step. A stronger update-rule method such as GDN2 may achieve similar gains over GDN/KDA without this 2× state expansion. I would like to see FVLA compared with GDN2 under the same training setup, reporting quality versus recurrent state size, decode-time state I/O, and throughput.
We have extended our experiments to GDN2 to investigate whether a stronger memory update rule can replace the benefits of FVLA. All models are trained under the same 140M/5B setting, using identical training conditions and evaluation protocols.
The following table compares GDN, GDN2, and their FVLA variants. State denotes the main recurrent matrix-state capacity relative to each native host.
| Model | State | Wiki ↓ | CS5 ↑ | Avg₆ ↑ |
|---|---|---|---|---|
| GDN | 1× | 45.50 | 40.17 | 15.05 |
| + FVLA (r = 1) | 2× | 45.36 | 39.96 | 17.36 |
| + FVLA (r = 4) | 5× | 43.60 | 40.41 | 19.33 |
| GDN2 | 1× | 42.82 | 39.43 | 16.48 |
| + FVLA (r = 1) | 2× | 43.24 | 39.94 | 18.85 |
| + FVLA (r = 4) | 5× | 41.92 | 40.30 | 21.73 |
The results reveal two complementary effects.
- A stronger update rule improves the native host. GDN2 achieves better language-modeling perplexity and retrieval accuracy than GDN without expanding the main recurrent matrix state. However, its Avg₆ retrieval accuracy (16.48) remains below that of GDN+FVLA-r1 (17.36).
- FVLA continues to provide additional benefits on GDN2. Adding FVLA-r1 improves GDN2's Avg₆ by 2.37 percentage points, while FVLA-r4 further increases the improvement to 5.25 points. FVLA-r4 also reduces Wiki perplexity from 42.82 to 41.92 and improves CS5 from 39.43 to 40.30.
These findings demonstrate that strengthening the memory update rule does not eliminate the benefits of quadratic response-space expansion. The two approaches improve different aspects of recurrent memory and can be combined to achieve further gains.
We have added the GDN2 comparisons to Section 4.2 (Table 1), with complete task-level results in Appendix D.2 (Table 9). The corresponding recurrent-state I/O and throughput considerations are addressed separately in our efficiency analysis.
[Part1] Response to Reviewer e4Jw
We appreciate this important suggestion. We have substantially extended our efficiency evaluation to include end-to-end training throughput, batched prefill and decoding throughput, peak GPU memory, and operator-level latency. In particular, we evaluate batch sizes of 1, 16, 64, and 256 to characterize efficiency beyond batch-one inference.
The following table summarizes the r=4 results on an RTX 5090. Inference throughput and peak memory are measured at batch size 256 and 2k context.
These results quantify the cost of expanding the recurrent state by 5×. While the additional computation reduces training and prefill throughput, FVLA retains 89.1% and 87.6% of native GDN and KDA decoding throughput, respectively, at batch size 256.
We also provide matched response-operator measurements, including forward, backward, and joint execution latency, together with the recurrent-state capacity accounting.
We have updated Section 4.7 (Figure 6) and added the complete measurements to Appendix I (Efficiency Analysis), including all evaluated batch sizes and FVLA configurations.