acceptodds
Under review as a conference paper at ICLR 2027

Function-Valued Memory: Query-Dependent Values for Linear Attention

Abstract

Matrix-state linear attention compresses history into fixed-size recurrent memory. Its design involves both update dynamics and the Query-to-Value function class represented by the fast state; under a matrix-scale budget, expanding the latter becomes a capacity-allocation problem. We introduce Function-Valued Linear Attention (FVLA), which generalizes the fixed Value vector associated with each memory coordinate to a Query-dependent response function: the Query determines both how strongly the coordinate is read and what Value it returns. A separable quadratic parameterization learns response directions across contexts while adapting their Value coefficients online within each context. Although nonlinear in the Query, FVLA remains linear in its online coefficients, preserving shared-error Delta updates and chunkwise computation. Across Gated DeltaNet and Kimi Delta Attention experiments spanning 140M and 340M configurations, FVLA improves language modeling and in-context retrieval. Across the four host–scale pairs, one and four response modes yield median relative gains of 10.5% and 24.0%, respectively, in average accuracy over the six retrieval tasks. Ablations show that nonlinear responses use additional fast-state capacity more effectively than linear state expansion. Response-mode scaling yields a clear quality-state-compute trade-off, with overall modeling quality improving as more capacity is allocated to the response space. Models trained at 2K context retain language-modeling gains through 512K-token streaming evaluation and improve long-range retrieval beyond training length. Together, these results identify response-space allocation as a key design dimension for finite-state associative memory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Linear Attention, Associative Memory, Fast Weights, Test-Time Regression, Recurrent Memory, Long-Context Modeling

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_e4JwOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes Function-Valued Linear Attention (FVLA), an augmentation to matrix-state linear attention. The method expands the read/write address of a Delta-style recurrent memory from dd to (r+1)d(r+1)d by adding learned separable quadratic features ψP(u)=u⊙Pu\psi_P(u)=u\odot Pu. The resulting recurrent state has size dv×(r+1)dd_v\times (r+1)d, so r=1r=1 and r=4r=4 correspond to 2× and 5× main fast-state size respectively. The authors instantiate FVLA on Gated DeltaNet and Kimi Delta Attention and report improvements in language modeling and in-context retrieval at 140M and 340M scales. The paper also includes ablations on normalization, widened linear states, random response directions, readout calibration, response-mode scaling, and long-context S-NIAH extrapolation.

Critical strengths
  1. Simple and modular construction.
    FVLA is easy to understand as an expanded feature/state space for Delta-style linear attention. It preserves the host’s recurrent update structure and can in principle be attached to GDN/KDA-like models.

  2. Consistent gains over the chosen hosts.
    At both 140M and 340M scale, $r=4 $ improves WikiText perplexity and retrieval averages over the corresponding GDN/KDA hosts. The paired training setup against GDN/KDA is a positive aspect.

  3. Useful ablations.
    The paper includes controls for write normalization, widened linear state, random quadratic directions, learned readout calibration, and full quadratic features. These help support the claim that nonlinear feature/state expansion is beneficial.

  4. Maintains fixed recurrent memory w.r.t. sequence length.
    FVLA increases the constant recurrent state size but does not introduce a KV cache or a memory that grows with context length.

Critical weaknesses

1. FVLA is a dense state/feature expansion, and the r=4r=4 efficiency tradeoff is under-characterized.
FVLA expands the recurrent address from dd to (r+1)d(r+1)d, so r=1r=1 and r=4r=4 use 2× and 5× main recurrent matrix state. All expanded blocks are densely read/written at every token, so the activation rate is effectively 100%. The paper reports useful throughput numbers for r=1r=1 and some operator-level latency scaling, but the strongest gains come from r=4r=4. I would like to see complete r=4r=4 end-to-end efficiency: prefill/decode latency, training throughput, peak memory, and preferably state-I/O or kernel latency. The batch-size-1 inference timing is useful for latency but does not fully characterize throughput-saturated serving.

2. The MoM-GDN comparison is present, but not controlled for activated state capacity.
The issue is not that MoM is absent: the paper includes a MoM-GDN public-checkpoint reference. However, MoM expands memory through routed states with sparse activation, while FVLA activates all expanded state blocks densely. Thus the comparison does not isolate total state, active state per token, activation ratio, state I/O, or throughput. Please clarify the MoM configuration and compare under matched activated state size or matched throughput.

3. GDN2 may cover FVLA’s r=1r=1 gain without doubling recurrent-state I/O.
FVLA r=1r=1 doubles the recurrent matrix state and roughly doubles fast-state I/O per recurrent decoding step. A stronger update-rule method such as GDN2 may achieve similar gains over GDN/KDA without this 2× state expansion. I would like to see FVLA r=1r=1 compared with GDN2 under the same training setup, reporting quality versus recurrent state size, decode-time state I/O, and throughput.

4. The polynomial feature alone is not isolated.
FVLA introduces nonlinear polynomial features together with expanded recurrent state capacity. I would like to see how much benefit the same polynomial feature gives to the GDN backbone at fixed state size, e.g. by replacing or compressing the address back to dimension dd. This would separate nonlinear feature-map gains from state-capacity gains.

5. Hybrid attention evaluation is missing.
All main experiments use pure GDN/KDA-style recurrent hosts. Since many competitive efficient LLMs use hybrid softmax/linear attention, and softmax layers may already improve retrieval, it is unclear whether FVLA remains beneficial in hybrid architectures.

6. Long-context evidence is encouraging but not broad enough.
The S-NIAH and streaming LM results are useful, but the long-context evaluation is still concentrated on synthetic needle retrieval. Gains also appear strongest on S-NIAH-1, while S-NIAH-2/3 remain more mixed.

7. The polynomial feature may provide gains without extra decoding I/O.
Related to point 4, if FVLA-style polynomial features improve GDN at fixed state size, then part of the reported gain may be achievable without doubling decoding state I/O. This ablation would be important for interpreting the efficiency benefit of FVLA.

8. Naive state expansion is only partially addressed.
The paper includes a small-scale 2× Wide GDN control, but the main results include r=4r=4, i.e. 5× recurrent state. I would like to see stronger naive expansion baselines at the main scale, such as larger key dimension or larger value dimension.

Questions for authors

Please respond to my concerns presented in the Weaknesses section. I would be happy to adjust my rating if these issues are well addressed.

11 comments
AuthorsOpenReview ↗

[Part1] Response to Reviewer e4Jw

Weakness 1: FVLA is a dense state/feature expansion, and the $r=4$ efficiency tradeoff is under-characterized.

I would like to see complete end-to-end efficiency: prefill/decode latency, training throughput, peak memory, and preferably state-I/O or kernel latency. The batch-size-1 inference timing is useful for latency but does not fully characterize throughput-saturated serving.

We appreciate this important suggestion. We have substantially extended our efficiency evaluation to include end-to-end training throughput, batched prefill and decoding throughput, peak GPU memory, and operator-level latency. In particular, we evaluate batch sizes of 1, 16, 64, and 256 to characterize efficiency beyond batch-one inference.

The following table summarizes the r=4r=4 results on an RTX 5090. Inference throughput and peak memory are measured at batch size 256 and 2k context.

ModelState (MiB)Training (K tok/s) ↑Prefill (K tok/s) ↑Decode (tok/s) ↑Peak (GiB) ↓
GDN1236.26197.188435.95.13
GDN + FVLA (r=4)6025.22116.977513.218.87
KDA1234.12161.357566.45.31
KDA + FVLA (r=4)6016.8480.146631.119.65

These results quantify the cost of expanding the recurrent state by 5×5\times. While the additional computation reduces training and prefill throughput, FVLA retains 89.1% and 87.6% of native GDN and KDA decoding throughput, respectively, at batch size 256.

We also provide matched response-operator measurements, including forward, backward, and joint execution latency, together with the recurrent-state capacity accounting.

We have updated Section 4.7 (Figure 6) and added the complete measurements to Appendix I (Efficiency Analysis), including all evaluated batch sizes and FVLA configurations.

AuthorsOpenReview ↗

[Part2] Response to Reviewer e4Jw

Weakness 2: The MoM-GDN comparison is present, but not controlled for activated state capacity.

Please clarify the MoM configuration and compare under matched activated state size or matched throughput.

Thank you for highlighting the distinction between sparse memory routing and dense response-space expansion. We have clarified the MoM-GDN configuration and added explicit measurements of total and active parameters, recurrent-state capacity, and training efficiency.

MoM-GDN uses four routed memories with top-2 selection and an additional shared memory. In contrast, FVLA expands the query-dependent response space within each recurrent memory, with all expanded state blocks actively updated.

ModelAvg₆ ↑Total / Active Params. (M)Resident / Active State (MiB)Training (K tok/s) ↑
GDN + FVLA (r=4)29.82354.37 / 354.3760 / 6025.22
MoM-GDN26.57512.35 / 411.29120 / 7211.38

The results show that FVLA achieves higher aggregate retrieval accuracy with fewer total and active parameters, smaller recurrent-state capacity, and higher measured training throughput. The corresponding logical state read/write volumes are 120 MiB and 144 MiB per token for FVLA and MoM-GDN, respectively.

Importantly, MoM and FVLA expand linear-attention memory along different dimensions: MoM selects among multiple memory banks, whereas FVLA enriches the response functions represented within each bank. These mechanisms are therefore complementary and could, in principle, be combined by incorporating FVLA into MoM's routed memories.

We have added the capacity breakdown to Appendix D.1 and expanded Section 4.7 (Figure 6) and Appendix I (Efficiency Analysis) with the corresponding training and inference measurements.

AuthorsOpenReview ↗

[Part3] Response to Reviewer e4Jw

Weakness 3: GDN2 may cover FVLA’s gain without doubling recurrent-state I/O.

FVLA doubles the recurrent matrix state and roughly doubles fast-state I/O per recurrent decoding step. A stronger update-rule method such as GDN2 may achieve similar gains over GDN/KDA without this 2× state expansion. I would like to see FVLA compared with GDN2 under the same training setup, reporting quality versus recurrent state size, decode-time state I/O, and throughput.

We have extended our experiments to GDN2 to investigate whether a stronger memory update rule can replace the benefits of FVLA. All models are trained under the same 140M/5B setting, using identical training conditions and evaluation protocols.

The following table compares GDN, GDN2, and their FVLA variants. State denotes the main recurrent matrix-state capacity relative to each native host.

ModelStateWiki ↓CS5 ↑Avg₆ ↑
GDN1×45.5040.1715.05
+ FVLA (r = 1)2×45.3639.9617.36
+ FVLA (r = 4)5×43.6040.4119.33
GDN21×42.8239.4316.48
+ FVLA (r = 1)2×43.2439.9418.85
+ FVLA (r = 4)5×41.9240.3021.73

The results reveal two complementary effects.

  1. A stronger update rule improves the native host. GDN2 achieves better language-modeling perplexity and retrieval accuracy than GDN without expanding the main recurrent matrix state. However, its Avg₆ retrieval accuracy (16.48) remains below that of GDN+FVLA-r1 (17.36).
  2. FVLA continues to provide additional benefits on GDN2. Adding FVLA-r1 improves GDN2's Avg₆ by 2.37 percentage points, while FVLA-r4 further increases the improvement to 5.25 points. FVLA-r4 also reduces Wiki perplexity from 42.82 to 41.92 and improves CS5 from 39.43 to 40.30.

These findings demonstrate that strengthening the memory update rule does not eliminate the benefits of quadratic response-space expansion. The two approaches improve different aspects of recurrent memory and can be combined to achieve further gains.

We have added the GDN2 comparisons to Section 4.2 (Table 1), with complete task-level results in Appendix D.2 (Table 9). The corresponding recurrent-state I/O and throughput considerations are addressed separately in our efficiency analysis.

Reviewer_ctxsOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a "function-valued memory" for linear attention (FVLA) that brings improvements in language modeling and in-context retrieval tasks across 140M and 340M parameter settings, where they mainly experimented with the Gated DeltaNet and Kimi Delta linear attention backbones. The idea is to turn the usual linear attention readout of

\begin{align} \mathbf{S}_t \mathbf{q} \rightarrow \mathbf{S}_t \mathbf{q} + \sum_i q_i \mathbf{C}_{ti} \mathbf{q} \end{align}

where the C term is defined so that in the end you get Stq+∑h=1rαhRt(h)ψPh(q)\mathbf{S}_t \mathbf{q} + \sum_{h=1}^r \alpha_h \mathbf{R}_t^{(h)}\psi_{P_h}(\mathbf{q}), something that is nonlinear in q\mathbf{q}.

The memory complexity grows from O(dv×d\mathcal{O}(d_v\times d to O((r+1)dv×d\mathcal{O}((r+1)d_v\times d, due to the additional rr features.

Their method can be interpreted as a feature expansion of the key and query, which they describe in setion 3.2. Because of this, no algorithm or code change is required to do the efficient chunkwise parallel computations as done in the original Gated DeltaNet and implemented in flash linear attention.

The experiments use a wide suite of benchmark datasets and they compare their models also to other baselines such as Transformer++ and GLA. It is clear that FVLA beats all the methods in main table (except transformers in retrieval tasks, which is well-known result for both SSMs and linear attention). Some ablation is also done on what drives the gains in the proposed method and some additional results on length extrapolation and long-range retrieval, which their method also performs favorably.

Critical strengths
  • It seems interesting that this parametrization of the key and query gives a clear performance boost. There should be some special properties that result from the construction of C, if the gain is so clear.
  • Results seem to support a clear gain in performance, in particular Table 1's FDA column, where the gain is massive over all baselines. Retrieval improvement is also visiblecompared to non-softmax attention architectures i.e. linear attention ones.
Critical weaknesses
  • The method seems like a simple expansion of the key and query features: It is slightly overselling the concept of functional-memory. What it really does is for each FVLA layer, introduce some P features that are fixed parameters, and R_t that is propagated across layers. I'm also slightly surprised, assuming that the wideGDN is parametrized correctly to ensure fair comparison, that simply expanding the key and query with some nonlinear features delivers such strong results. The reason is because the q, k, v are already nonlinearly constructed in practice (e.g. see the Kimi Linear technical report). It seems like it can technically absorbed into the shortconv + swish type of neural parametrization, and then enlarge the features, which would give you wideGDN.

  • Wide GDN Ablation: There are wideGDN experiments where the dimensions of k and q are expanded without the special features proposed in FVLA. There is a small ablation in table 2 to check whether the R and P features have some special significant effect on FVLA's expressiveness, but the comparison wideGDN is only compared for FVLA with r=1r=1 , it would be more valuable if r=4r=4 and larger values could be compared against. Then we can settle on whether this contribution is truly empirically useful or not

Questions for authors
  • In appendix D.3, line 1003-1004 it mentions "trainable parameter counts are closely matched by adjusting the feed-forward width". The feed-forward width actually has a major effect on any block because it enriches feature representations, adjusting it will definitely affect comparisons with other architectures. Could the authors explain why this is fair and whether it is possible to provide some results that do not modify the width?
8 comments
AuthorsOpenReview ↗

[Part1] Response to Reviewer ctxs

Weakness 1: The method seems like a simple expansion of the key and query features: It is slightly overselling the concept of functional-memory. What it really does is for each FVLA layer, introduce some P features that are fixed parameters, and R_t that is propagated across layers. I'm also slightly surprised, assuming that the wideGDN is parametrized correctly to ensure fair comparison, that simply expanding the key and query with some nonlinear features delivers such strong results. The reason is because the q, k, v are already nonlinearly constructed in practice (e.g. see the Kimi Linear technical report). It seems like it can technically absorbed into the shortconv + swish type of neural parametrization, and then enlarge the features, which would give you wideGDN.

We sincerely thank you for this insightful observation, which touches on a central motivation of FVLA and has helped us substantially strengthen our work. We would like to clarify two points:

  1. Our Wide GDN control already uses the suggested nonlinear feature construction. Following the FLA implementation, it employs the same short convolution and SiLU-based Key/Query construction as the original GDN host, while expanding the Key/Query dimensions.
  2. Expressive capacity does not necessarily imply ease of learning. We agree that sufficiently expressive nonlinear feature transformations could, in principle, learn FVLA-like quadratic interactions. However, such transformations must discover these interactions implicitly through optimization. FVLA explicitly learns the basis vectors of a quadratic response subspace, providing a structured inductive bias that may facilitate learning.

Motivated by your observation, we have substantially extended our Wide GDN ablations, including state-matched comparisons at the main 140M/5B scale in Section 4.4. We present these results in response to your questions below.

AuthorsOpenReview ↗

[Part2] Response to Reviewer ctxs

Question 1: In appendix D.3, line 1003-1004 it mentions "trainable parameter counts are closely matched by adjusting the feed-forward width". The feed-forward width actually has a major effect on any block because it enriches feature representations, adjusting it will definitely affect comparisons with other architectures. Could the authors explain why this is fair and whether it is possible to provide some results that do not modify the width?

To address these concerns, we have extended our comparisons to examine both parameter-matched and state-matched settings.

In our original 27M-scale ablations, FVLA introduced a small number of additional trainable parameters relative to native GDN. To control for this difference, we slightly adjusted the FFN width to closely match the total parameter counts, aiming to isolate the contribution of the quadratic response features. However, Wide GDN also introduces additional parameters by expanding the Key/Query dimensions. Applying the same parameter-matching strategy therefore requires adjusting its FFN width, introducing a potential confounding factor, as you correctly pointed out.

We therefore conducted additional state-matched comparisons at the main 140M/5B scale, keeping the FFN width fixed at 2,048 and the Value dimension fixed at 128. Under identical recurrent-state budgets, we compare two expansion strategies: increasing the linear Key/Query dimension (Wide GDN) and introducing quadratic response modes (FVLA). We evaluate both r = 1 (2× state) and r = 4 (5× state). The results are reported as improvements relative to native GDN.

StateExpansionΔ Params. (M)Δ Wiki ↓Δ Avg₆ ↑
2×GDN → Wide GDN+18.97−0.50+1.22
2×GDN → FVLA (r = 1)+1.57−0.14+2.31
5×GDN → Wide GDN+75.89−1.44+2.96
5×GDN → FVLA (r = 4)+6.29−1.90+4.29

The results reveal two important findings:

  1. FVLA achieves larger retrieval improvements under matched state budgets. At both 2× and 5× state capacity, allocating additional state to quadratic responses yields substantially greater retrieval gains than expanding the linear address dimension.
  2. FVLA achieves these gains with fewer additional trainable parameters. At 2× state capacity, FVLA improves retrieval more than Wide GDN, although Wide GDN achieves slightly better Wiki perplexity. At 5× state capacity, FVLA outperforms Wide GDN in both Wiki perplexity and aggregate retrieval accuracy, while introducing far fewer additional parameters.

These findings strengthen our original conclusion: explicitly learned quadratic responses provide a more effective allocation of recurrent-state capacity for contextual retrieval than simply widening the linear host.

We have updated the parameter-matched ablations in Section 4.3 (Table 2) and introduced the new state-matched comparisons in Section 4.4 (Table 3). Complete experimental configurations and evaluation results are provided in Appendix D.1 and Appendix D.3.

AuthorsOpenReview ↗

Thank you & Looking Forward to Further Discussion

Dear Reviewer ctxs,

We sincerely thank you for your thoughtful and constructive review. Your feedback has helped us clarify FVLA's underlying motivation and strengthen the experimental comparisons. We summarize our main responses and revisions below:

  • Quadratic Response Mechanism (W1). We confirmed that our Wide GDN control retains the original host's short convolution and SiLU-based feature construction.

  • Fair Comparison with Wide GDN (Q1). Following your suggestion, we conducted additional state-matched comparisons at the main 140M/5B scale, keeping the FFN and Value dimensions fixed. These experiments directly compare linear feature expansion with FVLA's quadratic response expansion under identical recurrent-state budgets.

  • State and Parameter Efficiency (W1, Q1). The new results show that FVLA achieves larger retrieval improvements with substantially fewer additional parameters at both 2× and 5× state capacity. At 5× state capacity, FVLA also achieves better language-modeling perplexity than Wide GDN.

We have incorporated these results into Sections 4.3–4.4 (Tables 2–3), with complete experimental details in Appendix D.

We sincerely appreciate your valuable suggestions, which have helped us strengthen both the empirical evidence and the presentation of our method.

Best regards,

The Authors

Reviewer_9e1aOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FVLA augments a recurrent associative memory with learned quadratic features, psi_P(q) = c[q ⊙ Pq], where the query is multiplied elementwise by Pq. Because the resulting readout depends nonlinearly on the query yet linearly on the online coefficient states, the method can use a normalized delta update driven by a shared error and can reuse the chunkwise computation of its hosts. The construction is evaluated on Gated DeltaNet and Kimi Delta Attention at the 140M and 340M scales. With four response modes, average retrieval and WikiText perplexity both improve across all four host/scale pairings, in exchange for extra recurrent state and extra computation.

Critical strengths

S1. The algorithm hangs together and is not especially complicated. Equations (9)–(12) spell out the equivalence to a linear fast learner operating in an expanded feature space, and Appendix C.1 explains why write normalization keeps the delta correction at its intended scale. What is useful here is the structured parameterization of the features and the way it is folded into delta memory; nonlinear response as such is not being introduced as a new idea.

S2. The experiments do more than set a larger memory against its unexpanded host. Table 2 and Appendix D.3 contain controls for write normalization, a wider linear memory, a joint-linear variant, fixed random features, a fixed readout gain, and a separate residual update, and these line up well with the response-space explanation the paper offers.

S3. Within the scales studied, the results have reasonable breadth. At r=4, Table 1 shows Avg6 rising from 24.96 to 29.82 for 340M GDN and from 25.56 to 30.09 for 340M KDA. Scaling of state and operators and end-to-end measurements are reported as well, so the paper does not lean solely on linear asymptotic complexity (Figures 3 and 5).

Critical weaknesses

W1. The most convincing causal control is run at a much smaller scale than the main models. The state-matched response-space ablations use roughly 26.82M parameters and 1.2B tokens, while the headline results come from 140M/340M configurations trained with seed 42 (Appendix D). Useful as the controls are, they do not show that the gain at the main scale beats what the same recurrent-state or compute budget would buy if spent on a stronger linear host. The main comparisons also lack repeated training seeds.

W2. The boundary between this work and nonlinear feature maps or polynomial sketches needs to be drawn more sharply. Product features and polynomial attention are acknowledged in Appendix A, and the paper’s own framing of FVLA is as a linear learner over learned quadratic features. The random-feature control helps, but a product/sketch control with an equal feature budget, run inside the same delta recurrence, would make it clearer what the specific q ⊙ Pq structure buys. The point concerns the mechanism; it is not a call for an expanded bibliography or for comparisons against work that appeared after the deadline.

W3. Claims about long context should stay conditional. That perplexity can be streamed through 512K tokens does not by itself show that information that far back is actually used. Table 9, moreover, has sizable regressions at r=1—140M KDA and 340M GDN among them—and at 512K the 340M r=4 S-NIAH-1 recall is just 7% and 17%. What the evidence supports is better behavior in particular regimes, r=4 above all, not long-range retrieval that is preserved across the board.

Questions for authors

Q1. Do you already have a state-matched linear-host comparison at the main scale, or results from additional seeds? For the main host/FVLA configurations, exact trainable parameter counts, recurrent-state bytes, and measured training cost would let readers compare the practical trade-off directly.

Q2. Which property of the learned quadratic feature accounts for its edge over a generic compact nonlinear feature map? Swapping in a bounded feature map within the existing ablation setup would be especially telling.

Q3. For the streaming perplexity experiment, do the scored target tokens stay the same as the length of available history is varied? It would also help to separate numerically stable streaming from a demonstrated benefit of distant context, and to address the retrieval regressions at r=1.

Q4. GDN is used in QWen and showed commercial success. However, this does not invalidate other state-space models. Do you have more results with other SSMs. Even for GDN, there are many enhanced versions.

Overall, I think this paper has interesting contribution points. Well polishing the paper mode will significanlty increase its acceptance chance.

6 comments
AuthorsOpenReview ↗

[Part1] Response to Reviewer 9e1a

Weakness 1: The most convincing causal control is run at a much smaller scale than the main models. The state-matched response-space ablations use roughly 26.82M parameters and 1.2B tokens, while the headline results come from 140M/340M configurations trained with seed 42 (Appendix D). Useful as the controls are, they do not show that the gain at the main scale beats what the same recurrent-state or compute budget would buy if spent on a stronger linear host. The main comparisons also lack repeated training seeds.

Question 1: Do you already have a state-matched linear-host comparison at the main scale, or results from additional seeds? For the main host/FVLA configurations, exact trainable parameter counts, recurrent-state bytes, and measured training cost would let readers compare the practical trade-off directly.

Thank you for highlighting this concern. We have extended the original 27M/1.2B response-space ablations to the main 140M/5B training setting, using the same training conditions and closely matched parameter budgets. We have updated the response-space ablations in Section 4.3 (Table 2) and Appendix D.3.1 (Table 10) with the complete 140M/5B results. The updated results are presented below.

VariantStateWiki ↓Avg₆ ↑
GDN1×45.5015.05
GDN + Norm1×46.7314.58
Wide GDN2×46.2916.73
Joint Linear2×46.2515.50
FVLA (Random)2×46.1416.55
FVLA (α = 1)2×45.4716.67
FVLA (ours)2×45.3617.36

These results reproduce the main findings of our original smaller-scale experiments under a parameter-matched setting. With closely matched trainable parameter counts and the same 2× recurrent-state budget, FVLA achieves the lowest Wiki perplexity (45.36) and the highest aggregate retrieval accuracy (17.36), outperforming the linear controls.

All paired main-model runs use seed 42. We did not run additional training seeds. Regarding random seeds, we follow the cross-architecture and cross-scale evaluation strategy adopted in recent linear-memory studies such as PDN [1] and MoM [2]. Specifically, we evaluate FVLA across multiple recurrent host architectures and model scales, consistently observing improvements across these settings.


Moreover, following your helpful suggestion to report exact trainable parameter counts, recurrent-state capacity, and measured training cost, we have added the following measurements for the 340M-scale GDN and KDA configurations.

ModelTrainable parametersState (MiB)Training (K tok/s) ↑
GDN341,791,1041236.26
GDN + FVLA (r=1)344,937,2162434.63
GDN + FVLA (r=4)354,374,4006025.22
KDA354,226,3681234.12
KDA + FVLA (r=1)357,372,4802426.38
KDA + FVLA (r=4)366,809,6646016.84

The recurrent-state capacities are measured per sequence in FP32, and training throughput includes forward, backward, and optimizer updates on an RTX 5090. We have incorporated these results into Appendix D.1 (Table 8) and Appendix I (Table 16), providing a direct accounting of FVLA's additional parameters, memory capacity, and training overhead.


References

[1] Tumma et al. Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences. ICML 2026.
[2] Du et al. MoM: Linear Sequence Modeling with Mixture-of-Memories. ICLR 2026.

AuthorsOpenReview ↗

[Part 2] Response to Reviewer 9e1a

Weakness 2: The boundary between this work and nonlinear feature maps or polynomial sketches needs to be drawn more sharply. Product features and polynomial attention are acknowledged in Appendix A, and the paper’s own framing of FVLA is as a linear learner over learned quadratic features. The random-feature control helps, but a product/sketch control with an equal feature budget, run inside the same delta recurrence, would make it clearer what the specific q ⊙ Pq structure buys. The point concerns the mechanism; it is not a call for an expanded bibliography or for comparisons against work that appeared after the deadline.

Question 2: Which property of the learned quadratic feature accounts for its edge over a generic compact nonlinear feature map? Swapping in a bounded feature map within the existing ablation setup would be especially telling.

Thank you for this insightful suggestion. To better isolate the contribution of FVLA's quadratic response structure, we have added two nonlinear-feature controls at the 140M/5B scale, using the same recurrent update rule and expanded-state budget:

  1. Learned Product: Motivated by polynomial-sketch feature maps [1], we use learnable quadratic product features ψprod(q)=c[(Aq)⊙(Bq)]\psi_{\mathrm{prod}}(q)=c[(Aq)\odot(Bq)], where both AA and BB are trainable. FVLA's feature ψP(q)=c[q⊙(Pq)]\psi_P(q)=c[q\odot(Pq)] is a structured special case with A=IA=I and B=PB=P, preserving the original query coordinates as one factor.
  2. SiLU: We replace the quadratic response with ψSiLU(q)=cSiLU⁡(Pq)\psi_{\mathrm{SiLU}}(q)=c\operatorname{SiLU}(Pq), providing a smooth nonlinear alternative to FVLA's homogeneous quadratic features.

The results are summarized below.

VariantParams.Wiki ↓Avg₆ ↑
Learned Product141.29M45.8517.31
SiLU139.72M45.5916.48
FVLA (Random)139.73M46.1416.55
FVLA (ours)139.72M45.3617.36

These results support the effectiveness of FVLA's structured quadratic response. FVLA achieves the best language-modeling perplexity and aggregate retrieval accuracy among the evaluated controls, even compared with the more flexible Learned Product features using additional parameters.

We attribute this advantage to FVLA's constrained quadratic parameterization, which preserves the original query coordinates while learning complementary response directions through PP. Together with the host's linear response, this structure enables FVLA to represent distinct response patterns within a compact recurrent state.

We have incorporated these additional controls and their complete results into Section 4.3 (Table 2) and Appendix D.3.1 (Table 10).


Reference

[1] Kacham et al. PolySketchFormer: Fast Transformers via Sketching Polynomial Kernels. ICML 2024.

AuthorsOpenReview ↗

[Part3] Response to Reviewer 9e1a

Weakness 3: Claims about long context should stay conditional. That perplexity can be streamed through 512K tokens does not by itself show that information that far back is actually used. Table 9, moreover, has sizable regressions at r=1—140M KDA and 340M GDN among them—and at 512K the 340M r=4 S-NIAH-1 recall is just 7% and 17%. What the evidence supports is better behavior in particular regimes, r=4 above all, not long-range retrieval that is preserved across the board.

We thank the reviewer for highlighting the limitations of relying on synthetic NIAH tasks to characterize long-context performance. Our original 512K evaluation was designed as an extreme length-extrapolation stress test to examine how far FVLA's effects on host retrieval could persist. At this maximum evaluated length (512K), retrieval accuracy deteriorates substantially across both native hosts and their FVLA variants. We also agree with the reviewer's observation that FVLA's improvements depend on the retrieval regime, with r=4r=4 generally providing broader gains.

To further evaluate the generality of these improvements, we have extended our experiments following the cross-domain language-modeling evaluation of CCQ [1]. Specifically, we evaluate six 340M models across six corpora and context lengths from 2K to 20K. FVLA (r=4r=4) consistently reduces full-window perplexity relative to both GDN and KDA across all 60 corpus-length combinations for each host, while r=1r=1 improves 56/60 and 55/60 combinations, respectively.

These additional results demonstrate that FVLA's benefits extend beyond synthetic retrieval to language modeling across diverse domains and context lengths. We have incorporated this evaluation into the revised Length Extrapolation and Long-Range Retrieval (Section 4.6), with detailed results in Appendix D.3.4.

Finally, the NIAH results reveal a clear rank-dependent pattern: although some r=1r=1 configurations exhibit regressions and retrieval accuracy declines at extreme context lengths, r=4r=4 provides broader and more substantial retrieval improvements across tasks and context lengths. Similar task-dependent variations are also observed in our native GDN/KDA baselines and in CCQ [1], PDN [2], and Mamba-3 [3], highlighting the limitations of synthetic retrieval as a standalone evaluation. To provide a more comprehensive assessment, we have retained the complete NIAH results (Appendix D.3.3, Tables 12–14) and complemented them with the cross-domain language-modeling and fixed-target PG-19 evaluations discussed above. We thank the reviewer for the constructive feedback.

Question 3: For the streaming perplexity experiment, do the scored target tokens stay the same as the length of available history is varied? It would also help to separate numerically stable streaming from a demonstrated benefit of distant context, and to address the retrieval regressions at r=1.

The original streaming experiment evaluates NLL on the final 2K tokens of each context window, without keeping the prediction targets fixed across history lengths. In addition to the cross-domain evaluation described above, we have conducted a fixed-target PG-19 evaluation following Simba [4]. Specifically, we hold the same 100-token prediction targets fixed while varying the available history, allowing us to directly examine the benefit of additional context.

The results show that FVLA can substantially mitigate performance degradation under extended histories. From 2K to 128K, native GDN's target NLL increases by 0.1567, whereas GDN-FVLA (r=4r=4) increases by only 0.0101.

We have added the fixed-target PG-19 evaluation to Appendix D.3.5, including the complete results and evaluation protocol. We thank the reviewer for highlighting this important distinction and helping us strengthen our long-context evaluation.


References

[1] Le et al. Don't Read Everything: A Curvature-Conditioned Query for Linear Attention. EMNLP 2026.
[2] Tumma et al. Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences. ICML 2026.
[3] Lahoti et al. Mamba-3: Improved Sequence Modeling using State Space Principles. ICLR 2026.
[4] Song et al. Sparsified State-Space Models are Efficient Highway Networks. TMLR 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers and ACs,

We sincerely thank all reviewers for their thoughtful evaluations, constructive suggestions, and continued engagement throughout the discussion. Your feedback has helped us substantially strengthen the experimental evidence and clarify the contributions and practical trade-offs of FVLA.

We are especially grateful that all three reviewers have raised their ratings: Reviewers 9e1a and ctxs from 2 to 3, and Reviewer e4Jw from 2 to 4. Your recognition and encouragement mean a great deal to us.

Guided by your suggestions, we have made several important improvements to the revised manuscript, summarized below.

  • Stronger Main-Scale Ablations and Fair Comparisons. Reviewers 9e1a, ctxs, e4Jw We extended our controlled experiments to the 140M/5B scale, including parameter-matched ablations and state-matched comparisons against Wide GDN with unchanged FFN width. The results demonstrate that FVLA achieves larger retrieval improvements with substantially fewer additional parameters (Sections 4.3–4.4, Tables 2–3).

  • Understanding the Source of FVLA's Gains. Reviewers 9e1a, ctxs, e4Jw We introduced additional nonlinear-feature controls and compact FVLA variants under fixed recurrent-state budgets. These experiments support the effectiveness of learned quadratic responses and clarify the trade-off between base feature capacity and response-space expressivity (Tables 2 and 4).

  • Generality Across Architectures. Reviewers 9e1a, e4Jw We extended our experiments to GDN2 and Mamba2, demonstrating that FVLA remains beneficial with different memory update mechanisms. Additional hybrid-attention experiments further show its compatibility with sliding-window softmax attention (Section 4.2, Table 1; Appendix H).

  • Broader Long-Context Evaluation. Reviewers 9e1a, e4Jw We added evaluations across six real-world corpora and ten context lengths, where FVLA-r4 improves perplexity in all 60/60 settings for both GDN and KDA. We also introduced fixed-target long-context evaluations and clarified the limitations of extreme-length retrieval (Section 4.6; Appendix D.3.4–D.3.5).

  • Comprehensive Efficiency Analysis. Reviewers 9e1a, e4Jw We supplemented the paper with detailed measurements of parameter counts, recurrent-state capacity, training and inference throughput, and GPU memory usage. We also clarified the active-state comparison with MoM-GDN, providing a more transparent assessment of FVLA's quality–efficiency trade-off (Section 4.7; Appendices D.1 and I).

  • Implementation and Presentation Improvements. Reviewers ctxs, e4Jw We clarified our FVLA-specific Triton optimizations, including fused quadratic feature construction and specialized cached decoding. We also refined the paper's methodological positioning, experimental discussion, and presentation to better communicate the contributions and limitations of FVLA.

We sincerely appreciate the time and care that all reviewers have devoted to our work. Your insightful suggestions have motivated meaningful improvements to both our experimental analysis and the revised manuscript. We believe these additions have made the work substantially clearer, more rigorous, and more convincing.

Thank you once again for your valuable guidance, support, and encouragement.

Warm regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.