- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The author proposes HiServe, aiming to address the memory cost of recurrent-state checkpoints in prefix caching for hybrid LLMs. It characterizes approximate low-rank structure across five recurrent and hybrid models, stores each layer/head state as truncated-SVD factors, and restores full-size states on reuse. An asynchronous GPU worker and two state pools integrate this representation into SGLang while retaining the existing model kernels. From the experimental results, the method achieved good throughput and TTFT benefits.
Critical strengths
- HiServe separates checkpoint storage from execution, enabling low-rank compression without changing existing model kernels.
- A useful empirical characterization of the storage–fidelity tradeoff. The paper shows that recurrent-state redundancy can be exploited while preserving predictions on the evaluated hybrid models. The author provides actionable evidence for choosing a state representation and show why compression should be evaluated through model outputs rather than reconstruction error alone.
Critical weaknesses
-
SVD’s practical advantage is unclear. Serving experiments compare only against uncompressed checkpoints. INT8 achieves similar GSM8K accuracy to rank-16 SVD, but their compression overhead and serving performance are not compared.
-
Quality and performance need validation together. The paper does not test quality after repeated checkpoint saves and restores during multi-turn serving. The reported serving gains also rely on one cache budget and fixed pool allocation, so their sensitivity to these choices remains unclear.
-
The authors do not clearly articulate the contributions of their work or the main problems and challenges in the setting they study.
-
The paper’s writing is cluttered with numerical details, which hurts readability.
-
There are still many writing errors and unclear data in the paper, which looks like a technical report.
Questions for authors
- Please compare rank-16 SVD with packed per-head INT8 under the same cache policy and memory budget, including staging memory. Report quality, throughput, and TTFT.
- Does Qwen3.5-4B maintain quality after repeated saves and restores through the serving cache? Does Table 2’s fidelity advantage hold with the deployed SVD settings—four oversampling dimensions and one power iteration?
- How sensitive are the serving gains to the cache budget and the memory split between full-state and compressed pools? Please report hit rate and TTFT for a few nearby configurations.
2 comments
Thank you for the additional experiment. The new experiment has basically solved my concerns, but considering the expression and innovation of the paper, I will maintain my score.
Thank you for the detailed review and the feedback on presentation.
W1 / Q1: SVD vs INT8 at equal budget, staging included. The full comparison of six formats is in our reply to Csmi. Every format receives the same cache memory, staging included. At about 4x, INT8 and rank-16 SVD hold a similar number of checkpoints and serve equally fast. SVD goes further, because its factors can themselves be stored at lower precision. Rank-16 SVD with BF16 factors reaches 8x and holds far more checkpoints than INT8, which raises the hit rate from 0.48 to 0.70 with no measurable accuracy loss. This is our recommended setting.
W2 / Q2a: Repeated saves and restores. The paper tests this in Table 3, where the state is compressed and restored repeatedly during generation on two 30B-scale hybrids, and task scores stay within two points. We added a harsher test on Qwen3.5-4B that compresses and restores the state before every decode step of GSM8K, which is hundreds of times per question.
Neither format differs significantly from the uncompressed model, at far more cycles than serving ever applies.
Q2b: Deployed settings. We reran the KL probes on identical inputs with the deployed solver settings, the paper's settings, and exact SVD. All three give nearly identical median KL and top-1 agreement, so the solver settings do not affect the results. Tables 3 and 8 already used the deployed settings.
W2 / Q3: Budget and pool split. We varied the cache memory and how it is divided between full-size and compressed checkpoints, at concurrency 16.
HiServe improves throughput and hit rate once the cache can hold the checkpoints that upcoming turns reuse. The gain remains at the largest setting, where the baseline already serves 42% of tokens from cache. The best split gives the full-size pool only what running requests need and the rest to compressed checkpoints. A larger full-size pool removes the gain. Appendices D and E.3 give the details.
W3–W5: Contributions. Our contributions are threefold. First, we show that recurrent-state checkpoints are approximately low rank across five models. Second, we design a compressed checkpoint format that is reconstructed on reuse, leaves model kernels unchanged, and composes with quantization to reach 8x. Third, we integrate it into SGLang's hybrid prefix cache and evaluate quality and serving at equal cache memory. The two challenges are that recurrent states cannot be sliced to earlier positions, and that compression must not delay prefill.
We hope the revision addresses your concerns and would be grateful if you would reconsider your assessment.