- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a compression method (FReS) for models using linear attention, by performing low-rank factorization of the recurrent state during prefill, and updating the factors during each decoding step with periodic recompression to ensure column size stay within a fixed budget. Experiments are run across three different hybrid models with tasks spanning long context evaluation and reasoning.
Critical strengths
- A training-free method that makes use of the low-rank state structure and proposes a training-free method that enables compression throughout decoding. Performs a wide range of experiments, which demonstrate noticeable serving benefit, reporting up to 1.9x throughput at high-concurrency.
Critical weaknesses
-
Experiments do not fully support claim that FReS preserves dense-state quality. The majority of tasks in RULER and LongBench have long prompts but only very short answers. This may explain why FReS matches baseline on RULER for all models as it keeps prefill fully dense and only compresses during decoding. Where generation is longer and requires more reasoning, the losses become more significant. The experiment setup also disables thinking mode, capping the output generation limit at 8K tokens, which is quite few for reasoning tasks. Long chain-of-thought reasoning, which these models are typically used for is largely untested. Since the paper's core contribution is supposed to be a favorable quality-efficiency trade-off, more emphasis should be placed on decoding where the compression occurs.
-
The evaluation baselines. The only baseline compared against is FIFO. The related works cite some other methods but neither are compared against. Since the states are stored in fp32, and FReS enables 2x savings in storage based on the setup, we could similarly achieve the same savings by using bf16 with no algorithmic change, and have more savings with additional quantization.
Questions for authors
- Currently, LongBench and RUELR are reported as aggregate results. Can you report the detailed results for each task within the benchmark?
- How does FReS work with longer output generation tokens and thinking enabled?
- How does FReS perform on challenging math reasoning benchmark, like AIME2026?
- Each recompression discards half of the factor columns and Table 2 shows state error increasing with decode length, with no signs of saturation. Could the authors report the state/output error over longer decoding (e.g 8K-32K generated tokens) to show whether the error plateaus or keeps growing?
We thank the reviewer for the constructive feedback. We agree that it requires particular attention to longer decoding, where repeated state compression takes place. We also add an INT8 quantization evaluation. Under our configuration, FReS reduces the memory footprint of recurrent state to 25%, approximately equivalent to the memory usage of storing the uncompressed state in INT8. Furthermore, we add quality evaluations of the INT8 quantization against FReS, as reported in Appendix D2.
We address the questions below.
Per-task results. We thank the reviewer for this suggestion. We provide task-level results in Appendix D4 to illustrate the effects of compression respectively.
The results show that FReS maintains quality close to the dense baseline on RULER and LongBench, with higher average scores than FIFO baseline across all models. On RULER, FReS matches dense baseline on most retrieval subtasks, with differences concentrated in a few information-tracking, question-answering, and aggregation tasks. Overall scores remain comparable or improve slightly. On LongBench, FReS exhibits small variations across models and subsets. FIFO, by comparison, shows more pronounced degradation on some subtasks. Overall, the quality changes introduced by FReS are localized to a small number of subtasks rather than reflecting widespread degradation.
Longer generation with thinking enabled and challenging mathematical reasoning. FReS works with longer output and thinking enabled by repeating more compression. To investigate this concern, we conducted additional evaluations on AIME 2026 and HMMT, increasing the generation budget to 32k tokens and set the thinking enabled.
As reported in Appendix D3, FReS outperforms the evaluated FIFO and INT8 baselines on these two benchmarks, but incurs noticeable degradation to the dense baseline in some settings, particularly for Qwen3.5-27B (AIME:23/30->17/30, HMMT:25/33->15/33).
We acknowledge that our earlier claim of dense-comparable quality was too broad, particularly for highly challenging reasoning benchmarks, but FReS still offers a more favorable quality–compression tradeoff. In comparison, the evaluated INT8 quantization baseline achieves less than 30% accuracy on both benchmarks across all three models. We will revise our claims to reflect these limitations.
State and output error over longer decoding. We have extended the evaluation to 32k tokens on WikiText-103 using an fp32 reference recurrence with a fixed rank of 32. As shown in Appendix D4, the recurrent-state error of FReS remains nearly constant from 4k to 32k tokens (0.447 to 0.444), while its output error increases from 0.198 to 0.262. These results provide evidence that recurrent-state error plateaus over the tested range, although output error continues to grow.
Our objective is to maintain approximation quality close to SVD-based recompression under a fixed rank budget while balancing accuracy and computational efficiency. At 32k tokens, FReS achieves state and output errors of 0.444 and 0.262, respectively, compared with 0.419 and 0.252 for SVD, indicating that the additional approximation error remains small even over long decoding sequences.