acceptodds
Under review as a conference paper at ICLR 2027

FReS: Factorized Recurrent State for Efficient Inference with Gated Delta-Rule Linear Attention

Abstract

Recurrent gated delta-rule linear attention has been widely adopted in recent hybrid language models, such as Gated DeltaNet (GDN) in the Qwen3 series and Kimi Delta Attention (KDA) in Kimi K3. By replacing the context-length-dependent key–value cache with a fixed-size recurrent state table, these architectures enable efficient long-context inference. However, these models maintains a dense recurrent state in each of its layers and heads for every active sequence. As the number of concurrent sequences increases, the replicated recurrent states lead to aggregate memory usage and traffic to grow linearly with concurrency, making GPU memory capacity and bandwidth critical bottlenecks for high-concurrency serving. To address these challenges, we propose FReS, a training-free inference method. Building on the observation that recurrent states exhibit low effective ranks, which indicates the potential for compression. FReS leverages low-rank matrix factorization to represent recurrent states compactly, reducing memory storage and traffic. FReS further employs an online, subspace-aware state maintenance to support incremental decoding while preserving most state insights and limiting error accumulation. Experiments show that FReS largely preserves dense-state quality while achieving up to the throughput of the dense-state baseline under high concurrency.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Recurrent Linear Attention, Gated DeltaNet, Kimi Delta Attention, Low-Rank Approximation, Efficient LLM Inference, High-Concurrency Serving

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PpiNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a compression method (FReS) for models using linear attention, by performing low-rank factorization of the recurrent state during prefill, and updating the factors during each decoding step with periodic recompression to ensure column size stay within a fixed budget. Experiments are run across three different hybrid models with tasks spanning long context evaluation and reasoning.

Critical strengths
  1. A training-free method that makes use of the low-rank state structure and proposes a training-free method that enables compression throughout decoding. Performs a wide range of experiments, which demonstrate noticeable serving benefit, reporting up to 1.9x throughput at high-concurrency.
Critical weaknesses
  1. Experiments do not fully support claim that FReS preserves dense-state quality. The majority of tasks in RULER and LongBench have long prompts but only very short answers. This may explain why FReS matches baseline on RULER for all models as it keeps prefill fully dense and only compresses during decoding. Where generation is longer and requires more reasoning, the losses become more significant. The experiment setup also disables thinking mode, capping the output generation limit at 8K tokens, which is quite few for reasoning tasks. Long chain-of-thought reasoning, which these models are typically used for is largely untested. Since the paper's core contribution is supposed to be a favorable quality-efficiency trade-off, more emphasis should be placed on decoding where the compression occurs.

  2. The evaluation baselines. The only baseline compared against is FIFO. The related works cite some other methods but neither are compared against. Since the states are stored in fp32, and FReS enables 2x savings in storage based on the setup, we could similarly achieve the same savings by using bf16 with no algorithmic change, and have more savings with additional quantization.

Questions for authors
  1. Currently, LongBench and RUELR are reported as aggregate results. Can you report the detailed results for each task within the benchmark?
  2. How does FReS work with longer output generation tokens and thinking enabled?
  3. How does FReS perform on challenging math reasoning benchmark, like AIME2026?
  4. Each recompression discards half of the factor columns and Table 2 shows state error increasing with decode length, with no signs of saturation. Could the authors report the state/output error over longer decoding (e.g 8K-32K generated tokens) to show whether the error plateaus or keeps growing?
1 comment

We thank the reviewer for the constructive feedback. We agree that it requires particular attention to longer decoding, where repeated state compression takes place. We also add an INT8 quantization evaluation. Under our configuration, FReS reduces the memory footprint of recurrent state to 25%, approximately equivalent to the memory usage of storing the uncompressed state in INT8. Furthermore, we add quality evaluations of the INT8 quantization against FReS, as reported in Appendix D2.

We address the questions below.

Per-task results. We thank the reviewer for this suggestion. We provide task-level results in Appendix D4 to illustrate the effects of compression respectively.

The results show that FReS maintains quality close to the dense baseline on RULER and LongBench, with higher average scores than FIFO baseline across all models. On RULER, FReS matches dense baseline on most retrieval subtasks, with differences concentrated in a few information-tracking, question-answering, and aggregation tasks. Overall scores remain comparable or improve slightly. On LongBench, FReS exhibits small variations across models and subsets. FIFO, by comparison, shows more pronounced degradation on some subtasks. Overall, the quality changes introduced by FReS are localized to a small number of subtasks rather than reflecting widespread degradation.

Longer generation with thinking enabled and challenging mathematical reasoning. FReS works with longer output and thinking enabled by repeating more compression. To investigate this concern, we conducted additional evaluations on AIME 2026 and HMMT, increasing the generation budget to 32k tokens and set the thinking enabled.

As reported in Appendix D3, FReS outperforms the evaluated FIFO and INT8 baselines on these two benchmarks, but incurs noticeable degradation to the dense baseline in some settings, particularly for Qwen3.5-27B (AIME:23/30->17/30, HMMT:25/33->15/33).

We acknowledge that our earlier claim of dense-comparable quality was too broad, particularly for highly challenging reasoning benchmarks, but FReS still offers a more favorable quality–compression tradeoff. In comparison, the evaluated INT8 quantization baseline achieves less than 30% accuracy on both benchmarks across all three models. We will revise our claims to reflect these limitations.

State and output error over longer decoding. We have extended the evaluation to 32k tokens on WikiText-103 using an fp32 reference recurrence with a fixed rank of 32. As shown in Appendix D4, the recurrent-state error of FReS remains nearly constant from 4k to 32k tokens (0.447 to 0.444), while its output error increases from 0.198 to 0.262. These results provide evidence that recurrent-state error plateaus over the tested range, although output error continues to grow.

Our objective is to maintain approximation quality close to SVD-based recompression under a fixed rank budget while balancing accuracy and computational efficiency. At 32k tokens, FReS achieves state and output errors of 0.444 and 0.262, respectively, compared with 0.419 and 0.252 for SVD, indicating that the additional approximation error remains small even over long decoding sequences.

Reviewer_LGDHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FReS, a training-free method for reducing the recurrent-state overhead of Gated DeltaNet (GDN) and Kimi Delta Attention (KDA) during LLM inference. The authors first analyze the spectral properties of recurrent states and observe substantial redundancy. Based on this observation, FReS maintains a factorized state using rank-one updates and periodic randomized subspace recompression, without reconstructing the dense state during decoding.

Experiments on three pretrained language models show that FReS generally retains competitive generation quality and improves serving throughput by up to 91% under high concurrency.

Critical strengths

1.The paper systematically analyzes the spectral properties of recurrent states across GDN- and KDA-based models. It shows that these states have low effective ranks across different context lengths and that preserving dominant spectral components is important for maintaining task accuracy. These findings provide useful empirical evidence for recurrent-state compression.

2.FReS combines factorized rank-one updates with randomized subspace recompression to maintain a compact recurrent state during decoding without reconstructing the dense matrix. The method requires no additional training and can be applied to existing pretrained models. Experiments on three language models demonstrate its practical effectiveness, achieving up to 91% higher throughput under high concurrency.

Critical weaknesses
  1. The core method combines standard low-rank factorization, rank-one updates, and randomized subspace approximation. Previous work has already investigated low-rank structure and state reduction in linear attention. The main contribution therefore appears to be an engineering adaptation of established techniques rather than a substantially new algorithm.

  2. Although the paper reports large throughput improvements, some reasoning benchmarks show noticeable accuracy degradation. For example, Qwen3.5-35B-A3B drops from 78.86 to 74.57 on MMLU-Pro and from 80.81 to 75.76 on GPQA-Diamond. These losses are not discussed in sufficient depth.

  3. The serving evaluation also lacks detailed measurements of actual GPU memory usage, achieved concurrency, and latency. Without these results, it is difficult to determine whether the reported throughput improvements provide a consistently favorable trade-off in practical deployment.

Questions for authors

Q1. How does FReS compare with existing recurrent-state compression methods, particularly structured pruning and mixed-precision quantization, under matched memory budgets? Can the authors provide accuracy and throughput comparisons?

Q2. FReS shows noticeable accuracy drops on MMLU-Pro and GPQA-Diamond for Qwen3.5-35B-A3B. What causes these degradations, and how sensitive are the results to the evaluation sample selection?

Q3. Table 3 only evaluates reconstruction errors under different rank budgets. Can the authors report task accuracy, GPU memory usage, and end-to-end throughput for different rank configurations?

Q4. The spectral analysis covers contexts up to 256K tokens, while the main quality benchmarks use much shorter contexts. Does FReS preserve generation quality at 32K–256K context lengths?

Q5. Table 2 shows increasing state and output errors as the decoding length grows. How does this affect generation quality beyond 2K decoding steps? Does the same behavior hold for the larger GDN and KDA models?

Q6. The paper reports theoretical recurrent-state memory reduction but does not provide end-to-end GPU memory measurements. How much GPU memory does FReS actually save, including the full-attention KV cache and other serving overheads?

Q7. The serving experiments mainly report throughput under unlimited request rates. Can the authors provide TTFT, TPOT, and tail latency under realistic request rates and different concurrency levels?

Q8. Section 4.3 states that the recurrent state is initialized at rank s, while Appendix A.3 specifies an initial rank of 32 followed by recompression to 16. Which configuration is used in the experiments?

1 comment

We thank the reviewer for recognizing the spectral analysis and practical serving benefits of FReS.

We agree that the individual mathematical methods are established. Our contribution lies in adapting and codesigning them into a training-free inference method for pretrained GDN- and KDA-based models. It preserves favorable quality and achieves up to 1.91× higher throughput under high concurrency.

Comparisons with structured pruning and quantization under matched memory budgets.

We conducted comparisons with structured Q/K pruning at 50% and 75%, as well as a plain INT8 quantization baseline, where the 75% pruning and INT8 are under memory budgets matched to FReS .

Results are reported in Appendix D2. The supplementary quality results for Qwen3.5-35B-A3B are:

MethodRULERLongBenchMMLU-ProGPQA-DiamondMATH-500GSM8K
Dense85.6933.0075.7180.8185.8096.80
FReS85.6932.8374.5779.2985.8095.20
FIFO72.8619.3166.8654.5574.2088.80
Structured pruning, 50%90.9635.6671.7166.1682.2092.40
Structured pruning, 75%10.596.445.434.552.201.20
INT8 recurrent state85.5832.4611.436.0680.001.20

These results show that FReS delivers better quality than the 75% pruning and INT8 baselines, while remaining competitive with the 50% pruning baseline.

We also measured end-to-end serving throughput for the pruning baselines. On ShareGPT at a client concurrency limit of 1,024, using two A800 80GB GPUs. The throughput is measured in output tokens/s. The results show that FReS outperforms the Pruning baseline at high concurrency.

ModelDenseFReSPruning 50%Pruning 75%
Qwen3.5-27B1,105.711,906.011,349.581,612.78
Qwen3.5-35B-A3B2,510.774,717.223,489.484,118.51

The INT8 implementation is not yet sufficiently optimized for a fair performance comparison.

Reasoning degradation and sensitivity to sample selection.

FReS is a lossy compression method, and the approximation errors it introduces can reduce accuracy on challenging reasoning tasks. As shown in Appendix D3, our additional evaluations on AIME and HMMT show more degradation in some settings, indicating that dense-level quality is not consistently preserved. The magnitude of this degradation may also depend on the difficulty and selection of evaluation samples.

Our objective is to achieve a better quality–efficiency tradeoff than the baselines. Despite these losses relative to the dense model, FReS outperforms baselines on quality evaluation while substantially reducing recurrent-state memory.

Quality beyond 2K decoding steps and behavior across larger GDN/KDA models.

To investigate this concern, we conducted additional evaluations on AIME 2026 and HMMT.

As reported in Appendix D3, FReS outperforms FIFO and INT8 baselines on these two benchmarks, but incurs noticeable degradation to dense baseline in some settings, particularly for Qwen3.5-27B (AIME:23/30->17/30, HMMT:25/33->15/33).

Our method can achieve dense-comparable quality in some testcases. But for highly reasoning benchmarks, FReS can still offers a more favorable quality–compression tradeoff. In comparison, the INT8 baseline achieves less than 30% accuracy on both benchmarks across models. We will revise our claims to reflect these limitations.

Due to hardware limits, we have not established whether larger GDN and KDA models exhibit the same trend.

Actual end-to-end GPU memory usage.

Since FReS improves serving throughput primarily by accommodating more requests under the same memory budget, the throughput gains observed in ShareGPT experiments also provide indirect evidence of improved memory efficiency.

To quantify this effect more directly, we additionally report the vLLM cache-capacity estimate for requests with an 8K-token maximum length. FReS increases the estimated capacity from 87.50 to 104.93 (+19.92%) for Qwen3.5-27B, and from 196.27 to 240.28 (+22.42%) for Qwen3.5-35B-A3B. Across all concurrency levels, the capacity gains range from 19.92% to 20.16% and from 22.42% to 22.89%, respectively. These show that FReS can accommodate more requests under a comparable memory budget, providing direct capacity-level evidence for its serving throughput benefits.

Initial rank and recompression configuration.

We thank the reviewer for pointing out this unclear description. In our experiments, we use R=32R=32 and s=16s=16, and the recurrent state is initially compressed to rank 16. We will revise it to make the configuration consistent throughout the paper.

We have not yet completed some experiments requested due to time constraints. We will conduct these experiments and add the results in a subsequent revision once they are available.

Reviewer_Eu6YOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed a lossy, training- and calibration-free inference method to compression memory states of gated delta-rule linear attentions. Motivated by the low-rank structure of recurrent states, FReS employs factorized rank-one updates and periodic subspace recompression to reduce memory overhead during decoding. Experiments on large-scale hybrid language models demonstrate largely preserved generation quality and up to 1.91$\times$ end-to-end serving throughput under high concurrency.

Critical strengths
  1. This paper is empirically strong, with substantial end-to-end serving improvements under high concurrency settings while staying training- and calibration-free, well-aligned with ICLR system track.

  2. The experiments are comprehensive and closed. The authors conduct extensive experiments on 27B–48B-scale models, covering downstream quality, kernel performance, and end-to-end serving with real conversation workloads. This provides convincing empirical evidence and strengthens confidence in the practicality of the proposed method.

Critical weaknesses
  1. The paper primarily builds on well-established observations and techniques, so its algorithmic novelty is limited. Its main contribution lies in practical system design and strong empirical performance.

  2. The method is lossy. While on current benchmarks the performance are fine, it does not indicate universality.

I view this work primarily as a practical systems contribution rather than an algorithmic one. I do not consider limited algorithmic novelty a major bottleneck towards acceptance in this setting. My main concern is whether the reported benefits generalize to realistic serving workloads and remain competitive with simpler alternatives, such as quantization.

Questions for authors
  1. How does FReS compare with or complement existing recurrent-state quantization methods?

  2. The serving evaluation primarily relies on ShareGPT workloads. Could the authors provide results on 1-2 additional representative real-world serving workloads?

  3. In hybrid models, the full-attention KV cache can also become a major memory bottleneck, especially for longer contexts. How does FReS integrate with existing KV-cache compression methods, and what are the potential combined benefits?

2 comments

We thank the reviewer for the positive assessment of our empirical evaluation and practical system contribution. We appreciate the questions and address them below.

Comparison with recurrent-state quantization. We thank the reviewer for raising this question. FReS and recurrent-state quantization reduce memory through different mechanisms: FReS exploits the low-rank structure of recurrent states, whereas quantization reduces the numerical precision of the state representation.

Under our configuration, FReS reduces the memory footprint of recurrent state to 25%, approximately equivalent to the memory usage of storing the uncompressed state in INT8. Therefore, we added quality evaluations of the INT8 quantization against FReS, as reported in Appendix D1. The results show that quantization leads to quality degradation on a fews of the evaluated datasets

In addition, the two approaches are potentially complementary: the low-rank factors maintained by FReS could be quantized, potentially yielding additional memory savings.

Generalization beyond ShareGPT workloads. We thank the reviewer for this suggestion. To extend our evaluation, we conducted additional streaming-serving experiments using a BurstGPT-derived workload on models. It provides request-length profiles collected from production LLM services, allows us to evaluate FReS under a different request composition and assess whether its serving benefits generalize beyond ShareGPT.

The results is reported in Appendix D1. At a concurrency limit of 1,024, FReS improve throughput by 52%, 24%, and 46%, respectively. These results provide additional evidence of FReS’s serving benefits beyond ShareGPT, particularly at high concurrency. The range of benefit is dependent on the model, concurrency, and metric.

Integration with KV-cache compression in hybrid models. We agree that the full-attention KV cache can become a substantial memory bottleneck as context length increases. FReS targets the recurrent states of the linear-attention layers, while KV-cache compression targets the full-attention layers. The two therefore are operated on different model layers and compute stages of serving, and are potentially complementary.

At the complementary design, FReS and KV-cache compression can be applied to their respective layers. Joint with compression, FReS could reduce memory footprint and potentially support larger serving batches or longer contexts within a fixed memory budget. The total benefit depends on the memory contribution of each component and the overhead introduced by the selected compression methods.

Thank you for the clarification, I will maintain my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.