acceptodds
Under review as a conference paper at ICLR 2027

WEAVE: Compressing State Changes Along Native Evolution in Linear Attention LLMs

Abstract

A new generation of hybrid linear-attention models replaces much of the growing KV cache with fixed-size recurrent states. Yet agent serving must retain many of these states as checkpoints for prefix reuse across turns and branches, creating substantial GPU-memory pressure and costly prefix recomputation under load. We present WEAVE, a training-free approach that encodes state changes from a shared anchor and efficiently constructs these compact representations from the model’s native updates. WEAVE combines shared-anchor residuals, recurrence-driven encoding from native model updates, and periodic compaction with fused GPU kernels for efficient online serving. We evaluate WEAVE on Qwen3.5-35B-A3B and Kimi-Linear-48B across reasoning, long-context, and agentic workloads. WEAVE compresses recurrent checkpoints by up to while maintaining task quality and outperforming INT8 and INT4 quantization on all six benchmarks. Integrated into vLLM's native prefix-cache path, the reduced state footprint lowers prefix eviction and recomputation under memory pressure, yielding up to lower TTFT and higher token goodput under SLO constraints.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Serving, ML Systems, Prefix Caching, Linear Attention, Recurrent-State Compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dKq2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the memory overhead of recurrent-state checkpoints in hybrid linear-attention LLM serving, where storing many checkpoints for prefix reuse causes cache eviction and expensive recomputation. It proposes WEAVE, which stores checkpoints as low-rank residuals from a shared anchor and constructs them from the model’s native state updates. Experiments on two hybrid linear-attention LLMs show up to 4.26× checkpoint compression with minimal quality degradation, along with up to 6.96× lower TTFT and 3.14× higher token goodput.

Critical strengths
  1. The paper addresses a practical problem in hybrid linear-attention LLM serving, where recurrent-state checkpoints consume substantial cache memory. WEAVE stores a shared anchor and low-rank residuals for subsequent checkpoints, constructed from the model’s native state updates.
  2. The evaluation simulates prefix-cache reuse by repeatedly restoring checkpoints and continuing inference. WEAVE achieves over 4× compression with little quality loss on both GDN (Qwen3.5) and KDA (Kimi-Linear) models, while maintaining higher task scores than the INT4 baselines.
  3. The implementation includes vLLM integration and optimized GPU kernels. End-to-end serving experiments show lower TTFT and higher goodput than uncompressed checkpoints at the same checkpoint density under memory pressure.
Critical weaknesses
  1. The paper does not compare against existing prefix caching methods for hybrid models, such as cache admission/eviction or checkpoint placement, which address the same memory pressure. The main baselines are only uncompressed checkpoints and simple quantization, so it is unclear how WEAVE compares to prior work.
  2. The quantization baselines are limited to basic per-column INT8 and INT4 schemes, so the comparison does not show whether WEAVE outperforms a stronger quantization method.
Questions for authors

Could the authors add comparisons with other baselines, such as existing prefix caching methods for hybrid models and stronger quantization schemes? This would make it clearer how WEAVE performs relative to existing approaches. A brief discussion of the concurrent work DASC, which also compresses GDN/KDA checkpoints, would also be helpful.

Minor: Some figures (e.g., Figures 1, 2, and 5) are hard to read because of their small fonts and dense panels. Please improve them in the future versions.

2 comments

Dear Reviewer, Just a gentle follow-up on our rebuttal. We've included additional experiments on stronger caching and quantization baselines. We'd be happy to hear your thoughts when you have a chance. Thank you!

Thanks for the additional experiments. The comparisons with Marconi, stronger INT4 baselines, and DASC help address my main concerns. I appreciate the authors' efforts and will keep my rating at 3.

Reviewer_FBBtOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how to compress linear attention state checkpoints for prefix caching. The authors find that state changes across checkpoints have low-rank structures. They then create a method that uses shared anchors and low-rank residuals to compress these checkpoints. The method also uses native model updates to avoid expensive SVD, and integrates efficient GPU kernels into vLLM. The result is around 4× compression with minimal performance drop, and significantly improving TTFT and throughput under high concurrency.

Critical strengths
  1. WEAVE is established on a real and interesting property of linear attention ckpts, and a practically useful method. Instead of compressing each state independently, the authors use the low-rank structure of state changes and use native model updates to avoid expensive computation.

  2. The author integrated their methods with vLLM and efficient GPU kernels also makes the method practical rather than just being theoretical.

  3. The experimental results are strong, as mentioned in the summary above, and generalizable, as they showed that the method works for both GDN and KDA.

Critical weaknesses
  1. Missing baselines: there are a variety of linear attention state compression methods, such as DASC [1], State Rank Dynamics [2], and State Reduction [3]. In particular, DASC also focuses on compressing recurrent-state checkpoints for prefix caching. I think comparing against these methods would make the experimental results stronger and help establish the advantages of WEAVE.

[1] DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving https://arxiv.org/abs/2608.30386 [2] State Rank Dynamics in Linear Attention LLMs https://arxiv.org/abs/2602.02195 [3] On State Reduction in Linear Attention https://arxiv.org/abs/2602.04852

Questions for authors

see the missing baseline

2 comments

Dear Reviewer, Just a gentle follow-up on our rebuttal. We've added the baseline comparisons you suggested and would appreciate your thoughts when you have a chance. Thank you!

Thank you for your response. I remain my positive score for the great work!

Reviewer_T6HhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WEAVE targets the persistent recurrent-state checkpoints that arise when serving hybrid linear-attention models, and compresses them. Checkpoints are kept as residual factors relative to a shared anchor; these factors are built from the model’s native state updates by way of sketches or bounded factors, and the scheme pairs periodic compaction with GPU kernels and prefix caching in vLLM. For Qwen3.5-35B-A3B and Kimi-Linear-48B, the authors report recurrent-checkpoint compression of about 4.1–4.3x with task scores close to native, plus substantial gains in TTFT and goodput when cache pressure is high. What is contributed is a representation tailored to the model together with a serving implementation—not a new, general algorithm for sketching matrices.

Critical strengths

S1. The motivation is sound. A live recurrent state of fixed size does not imply that keeping many reusable prefix checkpoints is cheap, and the shared-anchor design together with the bounded dependency speaks directly to concerns about random recovery and cache eviction (Sections 2–3).

S2. The codec draws on information the model already produces as it runs. Both the GDN sketch recurrence and the transport-aware factors for KDA are internally consistent, and the appendix covers rank calibration, publication checks, repair, and the retention of dependencies. The two-sided sketching primitive, which is not new, is cited as it should be.

S3. Both the fidelity of checkpoints and the behavior of the serving system are evaluated. Six tasks on two models appear in Table 1, and Appendix A.3 reports repeated restore-and-continue tests with 39–157 events per trajectory. Compression ratios are stated to account for anchors, convolution states, metadata, and repair payloads, and what is measured is an integrated serving path rather than offline SVD compression alone.

Critical weaknesses

W1. The size of persistent checkpoints is documented far more completely than the total online cost in resources. Appendix A.1 deliberately leaves sketches and projections out of the persistent byte count—fine for that particular ratio—but the peak live workspace of the encoder, and how it grows with the number of concurrent requests, goes unreported. In the same vein, the 3.51% interference figure is a single measurement of asynchronous compaction on Qwen, not the overhead of the full codec on both models (Appendix A.2). For the end-to-end efficiency being claimed, these distinctions are consequential.

W2. The large serving speedups call for a workload and a cache operating point that others could reproduce. Section 4.3 speaks of identical request streams and budgets, but the trace statistics, arrival and reuse characteristics, absolute cache allocation, and the summary statistic used for TTFT are not specified in enough detail to recreate the result. Fixing checkpoint placement is a helpful way to isolate the effect; still, a coarser operating point with exact checkpoints could trade recomputation for capacity, and the evidence so far does not show the benefit over that simple alternative at the same memory budget.

W3. How well quality is preserved, and how robust the calibration is, are only partly quantified. Appendix A.3 gives means over three runs with no spread and no paired uncertainty, and several of the Table 1 gaps from native behavior are small. Because the Qwen rank map is calibrated on just four inputs, RepoQA among them, the paper should state explicitly whether calibration and evaluation data are disjoint, and how often repair and fallback occur at the workload level. There is also a conflict to resolve: Section 4.4 describes the first errors in Table 2 as relative to the residual, while the table’s caption says every entry is relative to the state; until that is settled, the ablation cannot be compared quantitatively.

Questions for authors

Q1. Would it be possible to break down, per model, the persistent bytes, the peak live encoder/workspace bytes, the restoration time, and the complete cost of the codec—certificates, repairs, and fallbacks included? The compaction-only interference number for Qwen should be kept apart from the overhead of the full codec.

Q2. The serving trace, the cache settings, the statistic used to aggregate TTFT, and the uncertainty over repeated runs all need stating. Within the existing workload setup, could a same-budget comparison against less frequent exact checkpointing be provided?

Q3. Three clarifications would help: confirmation that calibration and evaluation data do not overlap, the variation in task scores and the repair/fallback rates across workloads, and a single consistent denominator for every reconstruction error in Table 2.

Q4. Theretical justifications of your proposed method would be helpful in better understanding its characteristic.

1 comment

Dear Reviewer, just a gentle follow-up on our rebuttal. We'd be grateful if you could take a look when you have a chance and let us know if any concerns remain. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.