acceptodds
Under review as a conference paper at ICLR 2027

HiServe: Low-Rank Prefix Caching for Hybrid LLMs

Abstract

Hybrid language models combine recurrent layers with full attention to achieve competitive model quality while reducing the growth of the key/value (KV) cache. However, reusing shared prefixes requires separate recurrent-state checkpoints at cached boundaries. Storing these checkpoints in full increases memory pressure, eviction, and repeated prefill. Across five recurrent and hybrid language models, we find that recurrent states exhibit approximately low-rank structure, enabling substantial compression through truncated SVD. HiServe performs this compression asynchronously on the GPU and reconstructs states on reuse. At rank 16, HiServe provides a 3.98× reduction in same-precision storage for the evaluated Qwen3.5 recurrent-state matrices, or 3.73× including unchanged convolution states. Separate evaluations, including long-context tasks and periodic recompression, show benchmark-score changes from -0.63 to +2.00 percentage points. On multi-turn ShareGPT reference-history replay with Qwen3.5-4B on an NVIDIA GB10, HiServe improves throughput by 11.5% and reduces mean time to first token by 47.9% over SGLang's uncompressed prefix cache at a matched 15.23 GB cache budget. Gains depend on cache allocation and budget. Complementary trace-driven simulations with Marconi show a mean token cache-hit-rate improvement of 20.0 percentage points on ShareGPT under equal cache-memory budgets. Low-rank recurrent-state compression can thus increase prefix reuse and improve serving performance.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM serving, prefix caching, hybrid language models, recurrent-state compression, low-rank approximation, state-space models, linear attention, memory-efficient inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mU2qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The author proposes HiServe, aiming to address the memory cost of recurrent-state checkpoints in prefix caching for hybrid LLMs. It characterizes approximate low-rank structure across five recurrent and hybrid models, stores each layer/head state as truncated-SVD factors, and restores full-size states on reuse. An asynchronous GPU worker and two state pools integrate this representation into SGLang while retaining the existing model kernels. From the experimental results, the method achieved good throughput and TTFT benefits.

Critical strengths
  1. HiServe separates checkpoint storage from execution, enabling low-rank compression without changing existing model kernels.
  2. A useful empirical characterization of the storage–fidelity tradeoff. The paper shows that recurrent-state redundancy can be exploited while preserving predictions on the evaluated hybrid models. The author provides actionable evidence for choosing a state representation and show why compression should be evaluated through model outputs rather than reconstruction error alone.
Critical weaknesses
  1. SVD’s practical advantage is unclear. Serving experiments compare only against uncompressed checkpoints. INT8 achieves similar GSM8K accuracy to rank-16 SVD, but their compression overhead and serving performance are not compared.

  2. Quality and performance need validation together. The paper does not test quality after repeated checkpoint saves and restores during multi-turn serving. The reported serving gains also rely on one cache budget and fixed pool allocation, so their sensitivity to these choices remains unclear.

  3. The authors do not clearly articulate the contributions of their work or the main problems and challenges in the setting they study.

  4. The paper’s writing is cluttered with numerical details, which hurts readability.

  5. There are still many writing errors and unclear data in the paper, which looks like a technical report.

Questions for authors
  1. Please compare rank-16 SVD with packed per-head INT8 under the same cache policy and memory budget, including staging memory. Report quality, throughput, and TTFT.
  2. Does Qwen3.5-4B maintain quality after repeated saves and restores through the serving cache? Does Table 2’s fidelity advantage hold with the deployed SVD settings—four oversampling dimensions and one power iteration?
  3. How sensitive are the serving gains to the cache budget and the memory split between full-state and compressed pools? Please report hit rate and TTFT for a few nearby configurations.
2 comments

Thank you for the detailed review and the feedback on presentation.

W1 / Q1: SVD vs INT8 at equal budget, staging included. The full comparison of six formats is in our reply to Csmi. Every format receives the same cache memory, staging included. At about 4x, INT8 and rank-16 SVD hold a similar number of checkpoints and serve equally fast. SVD goes further, because its factors can themselves be stored at lower precision. Rank-16 SVD with BF16 factors reaches 8x and holds far more checkpoints than INT8, which raises the hit rate from 0.48 to 0.70 with no measurable accuracy loss. This is our recommended setting.

W2 / Q2a: Repeated saves and restores. The paper tests this in Table 3, where the state is compressed and restored repeatedly during generation on two 30B-scale hybrids, and task scores stay within two points. We added a harsher test on Qwen3.5-4B that compresses and restores the state before every decode step of GSM8K, which is hundreds of times per question.

FormatGSM8KChange (95% CI)
Uncompressed85.6%n/a
SVD-1684.4%−1.21 [−2.57, +0.15]
SVD-16 + BF16 factors84.5%−1.06 [−2.39, +0.27]

Neither format differs significantly from the uncompressed model, at far more cycles than serving ever applies.

Q2b: Deployed settings. We reran the KL probes on identical inputs with the deployed solver settings, the paper's settings, and exact SVD. All three give nearly identical median KL and top-1 agreement, so the solver settings do not affect the results. Tables 3 and 8 already used the deployed settings.

W2 / Q3: Budget and pool split. We varied the cache memory and how it is divided between full-size and compressed checkpoints, at concurrency 16.

Cache memorySystemSlots (full-size + compressed)tok/sHit rate
13.58 GBBaseline96 + 0193.50.06
13.58 GBHiServe64 + 59192.90.06
15.23 GBBaseline128 + 0193.20.08
15.23 GBHiServe64 + 179213.80.49
19.36 GBBaseline208 + 0211.40.42
19.36 GBHiServe64 + 477229.40.75

HiServe improves throughput and hit rate once the cache can hold the checkpoints that upcoming turns reuse. The gain remains at the largest setting, where the baseline already serves 42% of tokens from cache. The best split gives the full-size pool only what running requests need and the rest to compressed checkpoints. A larger full-size pool removes the gain. Appendices D and E.3 give the details.

W3–W5: Contributions. Our contributions are threefold. First, we show that recurrent-state checkpoints are approximately low rank across five models. Second, we design a compressed checkpoint format that is reconstructed on reuse, leaves model kernels unchanged, and composes with quantization to reach 8x. Third, we integrate it into SGLang's hybrid prefix cache and evaluate quality and serving at equal cache memory. The two challenges are that recurrent states cannot be sliced to earlier positions, and that compression must not delay prefill.

We hope the revision addresses your concerns and would be grateful if you would reconsider your assessment.

Thank you for the additional experiment. The new experiment has basically solved my concerns, but considering the expression and innovation of the paper, I will maintain my score.

Reviewer_ENvMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HiServe, a low-rank recurrent-state caching system for hybrid LLMs that improves cache efficiency and serving performance through SVD-based compression while preserving inference quality.

Critical strengths
  1. Under a fixed hybrid cache budget, storing low-rank recurrent states allows more states to be retained, improving cache hit rates.
  2. At a low rank (e.g., rank 16), the proposed method achieves serving efficiency improvements while maintaining performance comparable to full-rank states.
  3. HiServe preserves compatibility with existing inference kernels by reconstructing compressed states into full-size states, requiring no modifications to the model's execution kernels.
Critical weaknesses
  1. Lack of reconstruction overhead evaluation. Reconstructing full-size states introduces additional latency on cache hits. However, the paper does not isolate this overhead, making it difficult to determine when reconstruction costs outweigh the benefits of increased cache reuse.
  2. Limited analysis of memory overhead. Although HiServe reports similar peak GPU memory usage under a fixed cache budget, asynchronous SVD introduces additional temporary memory requirements. The paper does not sufficiently evaluate how this overhead affects effective cache capacity and checkpoint evictions, particularly at higher SVD ranks or under tighter memory budgets.
Questions for authors
  1. Ambiguous legend in Figure 3. The legend does not clearly distinguish “SVD (curves)” from “Median (filled),” since both are represented using similar gray line samples.
  2. Higher SVD ranks reduce compression benefits while potentially increasing reconstruction overhead, which may outweigh the gains from additional cache hits. A rank-sensitivity study of TTFT and throughput would help characterize this trade-off.
  3. Given the fixed cache memory budget, how meaningful is the reported similar peak GPU memory usage? Particularly at higher SVD ranks, could the additional memory required for asynchronous compression reduce effective cache capacity and cause extra checkpoint evictions?
2 comments

Thank you for the careful review and the questions on overhead and memory.

W1: Reconstruction overhead. We measured reconstruction inside the server by timing a cache hit on a full-size checkpoint, a cache hit on a compressed checkpoint, and a cache miss.

Prefix length (tokens)Full-size hit TTFTCompressed hit TTFTMiss TTFT
25689.8 ms88.8 ms~118 ms
1,02492.2 ms91.5 ms~248 ms
4,09696.9 ms97.7 ms~960 ms
8,192102.7 ms100.8 ms~1,850 ms

Reconstructing a checkpoint takes 0.8 ms, which is under 1% of the time to first token. A hit on a compressed checkpoint is therefore as fast as a hit on a full-size one, and both are far faster than a miss.

Compression pays off by raising the hit rate. When it cannot, for example when the cache is too small to benefit or the baseline already holds most reusable checkpoints, throughput stays within noise of the uncompressed cache (§5.2, App. E.3).

W2 / Q3: Memory overhead. Thank you for raising this. We now measure it directly (§5.2, App. E.4). Compression does not reduce cache capacity, because pool sizes are fixed at startup and its staging buffers are already counted in every equal-memory comparison. Each SVD uses about 0.16 GB of temporary GPU memory, whatever the batch size. Measured end to end, peak GPU memory is slightly lower with compression than without it. The GPU linear-algebra libraries also allocate 0.78 GB once, when they are first loaded.

Notably, we don't recommend using SVD above rank 16, as that leads to slower SVD computation without much accuracy gain. SVD algorithms perform the best at lower ranks, and finding a good balance between accuracy and speed leads to the best performance here.

Q2: Rank sensitivity. A lower rank makes each checkpoint smaller, so more of them fit in the cache. The uncompressed cache reaches 193.4 tok/s and a hit rate of 0.08 in the same setting.

RankCompressed checkpointstok/sHit rateMedian KL (Table 7)
4570229.60.762.3e-3
8329226.30.718.2e-4
16179214.60.522.5e-4

Lower ranks serve better but change the model's predictions more. We therefore recommend rank 16 with its factors stored in BF16, which holds as many checkpoints as rank 8 with no measurable accuracy loss beyond rank 16. Ranks well above 16 make checkpoints too large to help.

Q1: Figure 3. Fixed. The INT8 lines now have their own dash patterns, and the legend names each series.

We hope these measurements address your concerns and would be grateful if you would consider them in your assessment.

Reviewer_ENvMOpenReview ↗

Comment

Thanks to the authors for their detailed response. Some of our concerns have been addressed. However, we are still unsure whether method provides meaningful benefits under sustained high GPU utilization and tight memory budgets. Therefore, we maintain our original rating of 2.

Reviewer_CsmiOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

For hybrid models (SSM / linear attention + attention), prefix caching needs a separate recurrent-state checkpoint at each cached boundary, since a recurrent state can't be sliced back to an earlier position. These checkpoints are large (51.5 MB each for Qwen3.5-4B in FP32), which limits how many prefixes can stay in cache. The paper observes that recurrent states are approximately low-rank (median effective rank 2-24 at 99% energy, vs 60-198 for random matrices of the same shape), and HiServe stores checkpoints as rank-r SVD factors, computed asynchronously on GPU with randomized SVD and reconstructed on reuse, implemented in SGLang. At rank 16 (~4x smaller checkpoints), quality changes are small (-0.6 to +2 pts on LongBench v2, GPQA, MMLU-Pro), and in serving on multi-turn ShareGPT, cache hit rate goes from 3% to 67%, with 11.5% higher throughput and 48% lower mean TTFT at the same cache budget.

Critical strengths
  • Important and timely problem: hybrid models (Qwen3.5, Nemotron-H, etc.) are becoming common, and prefix caching for them is much harder than for Transformers because of the large state checkpoints.
  • Simple and practical method: low-rank compression of the state, done asynchronously, with existing model kernels unchanged. Integrated into SGLang.
  • Clear empirical finding that recurrent states are low-rank, with a matched random-matrix control, across 5 models (Mamba2, Nemotron, Qwen3.5, DeltaNet, Gated DeltaNet).
  • Thorough and honest evaluation: next-token KL, downstream tasks on large hybrid MoE models (incl. long context and repeated compression), live serving, and trace simulations across workloads and cache policies. Limitations are stated clearly.
Critical weaknesses
  • Baselines for the storage format: states are stored in FP32, and the serving comparison is against uncompressed FP32 checkpoints. Simpler options like storing checkpoints in BF16 (2x) or INT8 (4x) are not compared in serving. In the KL experiments, per-head INT8 at the same 4x is better than SVD on 3 of the 5 models (pure DeltaNet / Gated DeltaNet), and 8-bit is on par with SVD on GSM8K. SVD + quantization is also not evaluated.
  • Serving evaluation is narrow: one model, one GPU (GB10), one cache budget and pool split, 8 concurrent requests, and a regime where the baseline hit rate is only 3%. It would help to see how the gains change with cache budget and load.
Questions for authors
  • How does HiServe compare in serving to storing checkpoints in BF16 / INT8, and to SVD with quantized factors?
  • How do the gains vary with cache budget, concurrency, and on a datacenter GPU? Does compression keep up under heavier prefill load (the paper notes sustained prefill can starve compression)?
  • What's the cost of the SVD itself?
1 comment

Thank you for the careful review.

W1 / Q1: Serving comparison with BF16 / INT8, and SVD with quantized factors.

All formats run in the same server at the same 15.23 GB budget, which includes HiServe's staging buffers. ShareGPT 10-turn replay, concurrency 16, 3 paired repetitions.

FormatSize reductionCheckpoints heldtok/sMean / p95 TTFT (ms)Hit rateGSM8K
FP32 (uncompressed)1x / 1x128193.2478 / 1,0560.0885.6%
BF162x / 2x157193.4474 / 1,0500.0885.1%
INT8 per-row3.9x / 3.6x238214.1332 / 7690.4884.0%†
SVD-164x / 3.7x243213.8323 / 7550.4984.4%
SVD-16 + BF16 factors8x / 6.9x393226.0255 / 4970.7084.5%
SVD-16 + INT8 factors15.5x / 11.6x622229.6234 / 4080.7683.7%‡

The reduction factors we measure here include the uncompressed convolution layer.

We see that the performance is similar between quantized and SVD techniques when the size reduction of the states is the same. Both of these techniques are cheap to run during serving, so they will both produce similar serving results. We note quantization + SVD leads to larger compression amounts, and better serving performance as well.

W2 / Q2: Budget, concurrency, datacenter GPUs.

SettingRepetitionstok/sMean TTFT (ms)Hit rate
Concurrency 4176.7 → 79.3452 → 3100.01 → 0.43
Concurrency 82129.2 → 137.2448 → 3140.04 → 0.45
Concurrency 163193.2 → 213.8478 → 3230.08 → 0.49

The arrow references the performance before and after enabling SVD compression. Concurrency refers to the number of concurrent requests.

We performed a small sweep over various numbers of concurrent requests amounts to provide a bit of context. We also simulated performance improvement of our compression through the Marconi simulator (Pan et al., 2025), which shows cache hit rate improvements over a sweep of cache budgets, workloads, and cache admission policies.

Q2 (heavier prefill load). Our scheduling policy gives prefill priority over compression, so compression starves when there is a lot of prefill. In multi-turn replay this does not happen, and more than 99% of eligible checkpoints are compressed. To show the starved case, we sent bursts of long prompts that each required a full prefill. The compression worker then had almost no idle time and compressed only 5.5% of checkpoints. Starvation costs compression, not latency or caching. Requests are never delayed, and the checkpoints that were not compressed stayed cached at full size and were all later reused. A smarter scheduling policy could let compression proceed under heavy prefill, but scheduling is not a main contribution of this work. Details are in App. E.4 and App. F.

Q3: SVD cost. Compressing a checkpoint takes about 11 ms and runs in the background, so it adds no latency to requests. Restoring a compressed checkpoint on a cache hit takes 0.8 ms, which is less than 1% of the time to first token. Each SVD also uses about 0.16 GB of temporary GPU memory, which does not take space from the cache (see our reply to ENvM).

Reference

Rui Pan, Zhuang Wang, Zhen Jia, Can Karakus, Luca Zancato, Tri Dao, Yida Wang, and Ravi Netravali. Marconi: Prefix caching for the era of hybrid LLMs. Proceedings of Machine Learning and Systems, 7, 2025.

Comments on the paper

1 comment

We thank the reviewers for careful and constructive reviews. Reviewers found the problem timely (Csmi), the method practical because it leaves model kernels unchanged (Csmi, ENvM, mU2q), and the storage–fidelity characterization useful (mU2q). The reviews asked for three things, and we ran all three:

1. Storage-format baselines in serving (Csmi, mU2q).

  • We added BF16, INT8, and SVD with BF16 or INT8 factors to the server and compared six formats at the same cache budget (App. C, E.3).
  • Low rank goes where quantization cannot. Rank-16 factors stored in BF16 give 8x over FP32 with no measurable GSM8K loss beyond SVD-16 itself. We argue that the SVD operation at rank 16 is nearly lossless, and can be applied on top of any quantization technique users wish to use, multiplying the compression.
  • Low rank is robust to repetition. Re-applied at every decode step, the paper's INT8 baseline (one scale per head, "per-head") loses 9.8 GSM8K points, while SVD-16 stays within noise. Per-row INT8 does not observe as large of a loss, but this highlights how fragile quantization is.
  • At the one point where both apply (4x, with FP32 factors), INT8 with one scale per matrix row ("per-row") serves as fast as SVD-16.

2. Overheads (Csmi, ENvM).

  • Restoring a compressed checkpoint on a cache hit costs 0.8 ms, a nearly insignificant amount.
  • Compressing one costs 11 ms and runs in the background.
  • The SVD's temporary workspace sits outside the cache budget. We now measure it: it is 0.16 GB at any batch size and cannot shrink the cache (§5.2, App. E.4).

3. Sensitivity (all reviewers).

  • We reran the live serving experiment while varying one setting at a time: the amount of GPU memory given to the cache, how that memory is divided between full-size and compressed checkpoints, the number of concurrent requests, and the SVD rank (App. E.3). These live runs add to the simulated sweep in §5.3.
  • The gain holds under heavier load and at a larger budget. At concurrency 16, the hit rate rises from 8% to 49% with SVD-16 and to 70% with BF16 factors.
  • The gain appears once the cache can hold the reuse working set, meaning the checkpoints that upcoming turns will look up.
  • This gives a simple sizing rule: keep the full-size pool no larger than running requests need.

Changes to the paper

  • §5.2: overheads, memory measurements, and a sensitivity summary with a new storage-format table (Table 5).
  • §4.4: revised INT8 framing.
  • App. C: storage-format and INT8-granularity paragraphs.
  • App. E.3–E.4: new sensitivity and overhead results (Tables 11–15).
  • App. D: an explicit sizing rule and a terminology list; serving terms are also defined at first use (§5.1).
  • Abstract and §1: one sentence each now states the sweep result.
  • Presentation: numerical detail moved from prose into tables, the appendix condensed (the five KL sweep tables are now one, Table 7), the Figure 3 legend fixed, and App. F expanded.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.