acceptodds
Under review as a conference paper at ICLR 2027

AlignPred: Error-Controlled KV Cache Compression via Alignment and Predictive Coding

Abstract

Long-running agents retain KV caches across turns and sessions for hours or days. Compressing these persistent caches requires reducing storage while controlling reconstruction error. We present AlignPred, a codec that combines local alignment with predictive coding under a configurable KV reconstruction-error budget. Local orthogonal transforms align related heads or layers into a common coordinate frame, allowing AlignPred to combine token-wise prediction with same-token prediction across heads or layers. Residuals are quantized on an error-bounded lattice and entropy-coded, without learned predictors, encoder-side mode search, or large joint transforms. Across three attention mechanisms, AlignPred requires less storage than the evaluated baselines at comparable measured reconstruction error. Against the evaluated CPU codec implementations, it decodes at least 1.6× faster while matching or improving encode time. AlignPred achieves near-native task scores at substantially compressed operating points on LongBench-Pro at 128K context. AlignPred's SGLang HiCache integration reduces persistent target-KV storage by 42–72% relative to native HiCache, with comparable latency.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache compression, persistent KV storage, predictive coding, error-controlled compression, LLM serving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Wda2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
3(+1)
Read review
Paper summary

The paper proposes AlignPred, an error-controlled KV-cache compression method designed to reduce persistent KV-cache storage. Experimental results demonstrate promising trade-offs between storage rate and downstream task performance, particularly at lower storage rates.

Critical strengths

The motivation is supported by useful illustrations. Figures 2 and 3 help demonstrate the potential benefits of exploiting cross-block dependencies through alignment-based prediction, while Figure 1 provides context for persistent KV-cache storage.

Critical weaknesses
  1. Unclear research motivation and limited methodological novelty. The research question and its distinction from existing methods, such as KVTC and KVComp, are not sufficiently articulated, making the proposed approach appear to be an incremental combination of established techniques. This concern is further amplified by the unclear organization and transitions in the Introduction. For example, the discussion moves from error-controlled KV-cache compression to predictive coding without clearly explaining the connection between these concepts. Similarly, the discussion in lines 72–85 is not sufficiently connected to the preceding motivation.

  2. Limited or unclear improvements from the proposed mechanism. Although Figure 8 demonstrates near-native downstream performance at reduced storage rates, it remains unclear to what extent these improvements can be attributed to the proposed alignment-based prediction mechanism. In Table 2, the incremental storage reduction from inter-block prediction is relatively small, particularly for DeepSeek, where the improvement is less than 1%. Can the authors explain why the benefit varies across different models and attention architectures? Furthermore, the main takeaways from Section 5.4 are not sufficiently emphasized.

Questions for authors
  1. Can you explain the specific limitations of existing approaches and explain how the proposed alignment-based prediction addresses them? (Improve current presentation in the introduction for clear definition of research question and why proposed methods are needed).
  2. My last two questions in weakness point 2.
3 comments
AuthorsOpenReview ↗

W1 / Q1 Research motivation and methodological contribution && W2 / Q2 Gain attribution and model-dependent effects

W1 / Q1 Research motivation and methodological contribution

Thank you for highlighting this gap in the presentation. Our research question is: how can we exploit dependencies across KV heads or layers to reduce persistent storage under an explicit reconstruction-error budget?

This question arises from a practical deployment need: application teams are often reluctant to adopt lossy KV compression without explicit numerical-error control. The error budget specifies the acceptable distortion, but does not determine how efficiently the cache is represented within that budget. AlignPred addresses this second problem through alignment-conditioned predictive coding.

The key observation is that related heads or layers may contain correlated information expressed in different coordinate systems. Direct prediction cannot fully exploit this structure when corresponding coordinates are misaligned. We therefore learn compact local rotations, align the representations, and predict each block from previously reconstructed related blocks. Error-controlled residual coding then encodes what the prediction does not capture. This explains the connection between error control and predictive coding: error control sets the fidelity constraint; alignment and prediction reduce the storage required under that constraint. Table 2 supports the importance of this interaction: cross-block prediction increases storage without alignment but reduces it after alignment.

The distinction from the cited methods is also important. KVTC uses PCA-based joint transforms to concentrate information before quantization; AlignPred uses local coordinate alignment to expose dependencies for predictive residual coding. KVComp targets GPU-resident caches and fused decompression–attention computation, where prediction dependencies introduce execution overhead. AlignPred targets persistent storage, allowing predictive reconstruction during cache restoration, outside the attention kernel.

Our contribution is the complete method that connects these choices: reusable local alignment, closed-loop cross-block prediction, error-controlled residual coding, and native cache save/restore integration. We will reorganize the Introduction around the deployment motivation, the unresolved KV-structure problem, and this design rationale. We will also rewrite lines 72–85 to connect each design choice directly to the research question.

W2 / Q2 Gain attribution and model-dependent effects

AlignPred is a complete compression method whose components contribute jointly to the overall storage reduction. Section 5.4 isolates their contributions: alignment makes cross-block prediction effective, while intra-block prediction and residual coding provide complementary savings. The small cross-block increment is therefore an additional benefit, rather than the overall gain of AlignPred.

Its magnitude varies with the available KV correlations. DeepSeek’s native CSA/C4 compression may leave less cross-layer redundancy to exploit; Appendix Figure 12 shows weaker aligned inter-layer same-bit mutual information than in GLM. This helps explain its smaller cross-block gain. We will make the component contributions and model-dependent behavior clearer in Section 5.4.

AuthorsOpenReview ↗

Thank you for your constructive feedback

Thank you for your constructive feedback. We have clarified the research motivation, methodological contribution and component attribution, including the model-dependent results. Please feel free to ask any follow-up questions.

Thank you for your response. I don't have any further concerns. I raised presentation and soundness score.

Reviewer_phr5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper studies compression of persistent KV caches, where the caches retained across sessions for hours or days and resorted on cache hits. This setting is distinct from GPU-resident compression. AlignPred aligns groups of related cache blocks (neighboring heads, or layers of a shared latent cache) into a common coordinate frame via per-block means and orthogonal rotations fitted by Generalized Procrustes Analysis on calibration data. After alignment, a fixed prediction schedule codes the first block of each group from its neighbors and later blocks from earlier already reconstructed blocks at the same token and channel. Residuals are quantized on an error-bounded latticewhose step follows a user specified relative reconstruction error budget, with fp32 escapes, and entropy encoder rANS. Across three attention mechanisms (GQA, MLA latent, CSA latent) the codec stores less than the evaluated baselines at comprable measured error, decodes at least 1.6x faster than evaluated CPU implementations, and preserves near vanilla scores for LongBench Pro at 128k, and further, integrated with SGLang HiCache, redices persistent target KV storage by 42-72% with comparable restore latency.

Critical strengths
  • The problem is timely and relevant. Persistent KV retention is supported by commercial APIs with retention windows from hours to days (given in Appendix E), and the restore once property in this setting is correctly identified as the opening that makes predictive coding affordable (unlike GPU-resident compression where predications where prediction conflicts with fused decompression).

  • Error control by construction - The residual step eta=eps∗∣∣X∣∣F/sqrt(n)eta = eps*||X||_F/sqrt(n) with a per-value lattice check whihc gives a relative reconstruction error bound in exact arithmetic. The orthogonal alignment is what carries the guarantee back to the original coordinates. The reported errors are measured, including finite-precision effects.

  • Complete specification - Alg.1 + appendices suffice to reimplement. The prediction schedule, escape path, serialization, GPA fitting, grouping, calibration and all baseline configurations. Sensitivity studies show the blending weights and group partitions are not load-bearing.

  • Genuine system evidence. A 36-point serving matrix (3 models x3 context lengths x 4 batch sizes). Negative results are reported plainly (Deepseek concurrency regression, storage reduction does not implies lower host memorym cache specific calibration loses on two of three models).

  • Honest accounting - Bits per value includes the oded metadata and headers. Reusable calibration parameters are excluded but disclosed and separately costed. The error-to-quality relation is acknowledged to be imperfectly monotonic.

Critical weaknesses
  1. The central know is not shown to do what the motivation promises. The paper's stated reason for error-controlled compression is that reconstruction-error budget gives a transferable fidelity constraint for unseen workloads (fig. 11). But Appendix A concedes the bound does not guarantee downstream task quality and that per-workload task-level validation remains necessary, and Section 5.3 reports the error-to-score relation is only (imperfectly) monotonic, The experiment that would establish the claim is select the error target on one workload or domain and show task quality is preserved on others is missing. As written, the error budget is a well behaved rate knob rather than the validation shortcut that the motivation implies. Further, the user has no guidance for choosing the budget for a new application, which is the practical form the same gap.

  2. Baselines are adapted reimplementations. The KVCpdec and KVTC-style references are the authors' own adaptations (software HEVC; joint PCA with deflate), not official implementations. The margin over the strongest baseline (Rotation + SZ3) is roughly 0.4-0.5 bits per value (about 9%), so implementation quality is load-bearing for the rate-distortion claim.

  3. the cross-block contribution of alignment is small relative to its prominence. The ablation shows inter-block predcition changes storage by -0.04 to -0.17 bpv after alignment (and hurts without it). So most of the gain comes from the intra-block prediction and the lattice coder. The paper nerrative centers alignment-conditioned cross-block predicion while the measured effect of that component is the smallest in the pipeline. It would be clearer if the breakdown were stated as directly in the main text.

  4. The method depends on an untested structural assumption. A single fixed rotation per block aligns blocks only to the extent that different heads' or layers' projections are approximately rotation-related, and the paper neither states nor measures this assumption directly. The frozen rotation variant is also the only viable one (cache-specific calibration loses on two out of three models), so the question of how well one calibrated rotation bank transfer across domain matters, and only five calibration domains are evaluated.

  5. The serving benefit is conditional. The latency improvements hold at batch size 1 and most concurrent loads. Deepseek, however, falls behind under concurrency (sync. restore on the scheduler thread). The 42-72% storage reduction is relative to native HiCache file storage which is fp16 for two of the three models, against already fp8 Deepseek the gain is 42% and the latency edge largely disappears.

  6. Scope - GPU resident is out of scope, and near native task quality holds across the full rate range only on Qwen. on GLM and Deepseek it holds at selected operating points. The evaluation covers one storage tier (three models and five content domains).

Questions for authors
  1. Does the error target transfers across workloads? If you select eps on one domain (say English retrieval), is task quality preserves at the same eps on other domains (code, mathematics)? This experiment would directly support the motivation in fig. 11, and the data for it appears to be collected already.

  2. How should one pick the eps for new application in practice? Is there any mapping, even rough, from error budget to expected task degradation that does not require running the full task evaluation?

  3. How well does a single fixed rotation per block captures cross-block structure? what fraction of the cross block predictability does GPA alignment recover relative to the best per-pair or per-token map? does the auxiliary rotation bank degrades on domains far from the five calibration domains?

  4. why is the cross-block gain after alignment small (0.04-0.17 bpv)? Is the residual dependence non-linear, token-dependent in a way one rotation cannot capture? or already covered by intra-block prediction?

  5. Can you report the main rate-distortion comparison against the official KVTC implementation (or explain why the adapted reference is faithful)? How sensitive are the margins to the baseline adaptations?

  6. Under concurrency, what is the restore-latency crossover point at which AlignPred's larger decode work overtakes its smaller transfer (the DeepSeek case)? Is an asynchronous restore path feasible?

== minor comments ==

  • The rate-control multiplier "m" used throughout the results tables is never defined in one place; state the reference error it multiplies and where it is calibrated.
  • Define "block", "group", and "object" once, early; the terms are used in the abstract and introduction before the definition in Section 3.2, and the geometry table is in Appendix B.3.
  • The introduction's use of "prediction" and "prediction references" (e.g., around the discussion of GPU-resident methods) is hard to parse before the method is presented; consider "cross-block prediction of aligned cache values" at first use.
  • The sentence on Jiang et al. finding that "a prediction stage helps compression but conflicts with that fusion" needs rewriting.
  • Keep the "same-bit" qualifier on every MI claim. The sentence "Alignment also increases this cross-head MI" can be misread as total mutual information, which an orthogonal transform cannot change; one sentence noting that alignment redistributes dependence into coordinate-aligned bit planes would prevent this reading.
  • The INT8 codes in Figure 3 are described as diagnostic, but the choice of 8 bits and of a scalar quantizer is not justified; a sentence on why this diagnostic reflects the codec's regime (or a quantizer-free MI estimate) would help.
  • "High-order bits" is defined only by the figure axis.
  • Equation (3): state that the minimization is joint over the rotations and the template M, and point forward to the alternating solver in Equation (10).
  • Consider citing Kampeas and Haleva, "Joint Encoding of KV-Cache Blocks for Scalable LLM Serving" (arXiv:2601.03067), which exploits cross-block KV redundancy from a different direction (cross-request block fusion for serving memory rather than per-cache coding for persistent storage).
  • Table 1's "ties Rotation + SZ3 on encode" corresponds to a 2-4% encode deficit; say "within 5%" or report the ratio.
8 comments
AuthorsOpenReview ↗

W1 / Q1 / Q2 Error budget selection and transfer && motivation

W1 / Q1 / Q2 Error budget selection and transfer

Thank you for highlighting this ambiguity. We apologize for not clearly explaining our motivation. In practical deployment, application teams are reluctant to adopt lossy KV-cache compression without explicit error control: tasks differ in their sensitivity to error, so scores on existing tasks do not establish acceptable distortion for a new application. We therefore introduce ε as an explicit, adjustable reconstruction-error budget, allowing compression settings to match the numerical fidelity already accepted by the application. This does not imply that the same error budget is universally transferable across tasks, as clarified in lines 42–44 of the paper.

Applications may already have validated FP8, INT8 or FP4 KV-cache quantization. AlignPred can match the error budget of that accepted configuration. Our comparisons support this practical starting point: codecs achieve similar task scores at comparable errors in the tested low-to-moderate-error range . This similarity is particularly evident in the near-lossless regime.

Near-matched NRMSE comparisons across three tasks and six reference configurations.

HotpotQA (20 inputs; Native F1 71.810)

ReferenceNRMSE % Ours / baselineF1 Ours / baselineTotal bpv Ours / baseline
TurboQuant-MSE 2/235.962 / 34.29069.275 / 72.9761.216 / 2.423
TurboQuant-MSE 3/318.410 / 18.65071.810 / 71.3102.124 / 3.401
TurboQuant-MSE 4/49.834 / 9.77671.810 / 71.8102.935 / 4.380
KIVI 8/80.498 / 0.50672.810 / 72.8107.265 / 9.150
KIVI 4/48.287 / 8.39671.810 / 72.8103.165 / 5.236
KIVI 2/240.732 / 41.15574.275 / 71.8101.031 / 3.278

MultiFieldQA_en (15 inputs; Native F1 61.231)

ReferenceNRMSE % Ours / baselineF1 Ours / baselineTotal bpv Ours / baseline
TurboQuant-MSE 2/235.852 / 33.97359.546 / 57.4831.530 / 2.653
TurboQuant-MSE 3/318.271 / 18.49156.373 / 57.5642.424 / 3.615
TurboQuant-MSE 4/49.752 / 9.70060.638 / 59.2663.223 / 4.577
KIVI 8/80.494 / 0.50161.231 / 61.2317.479 / 9.266
KIVI 4/48.218 / 8.31260.957 / 60.6603.450 / 5.417
KIVI 2/240.656 / 40.51658.247 / 58.4531.346 / 3.493

Qasper (20 inputs; Native F1 51.762)

ReferenceNRMSE % Ours / baselineF1 Ours / baselineTotal bpv Ours / baseline
TurboQuant-MSE 2/235.407 / 33.69046.594 / 45.3261.773 / 2.882
TurboQuant-MSE 3/318.107 / 18.37252.664 / 52.2612.649 / 3.828
TurboQuant-MSE 4/49.673 / 9.66551.732 / 53.1023.434 / 4.774
KIVI 8/80.490 / 0.49751.960 / 51.7627.619 / 9.382
KIVI 4/48.151 / 8.25852.304 / 52.5323.656 / 5.600
KIVI 2/240.116 / 40.23245.058 / 45.2631.594 / 3.708

These comparisons on 55 Qwen inputs support reconstruction error as a common fidelity reference in the tested low-to-moderate-error range.

To directly address your concerns about transferability, we conducted a set of experiments. we selected the budget on English retrieval and reused it unchanged on code and mathematics. All three budgets scored 100 on 200 source inputs; the source-task rule therefore selected the largest, εcontrol\varepsilon_{\mathrm{control}} = 0.107. We then froze it for 300 code-debugging and 300 mathematics inputs.

English-retrieval budget transfer to code and mathematics

TaskNative scoreFP8 scoreFrozen AlignPred scoreAlignPred total bpv
English retrieval100.000100.000100.0003.814
Code debugging58.00058.00057.3332.717
Mathematics49.33349.66749.0002.291

These results also show that sensitivity to the error budget varies across tasks: the same budget preserves the retrieval score but causes small score decreases on code and mathematics. Budget selection should therefore account for the target task’s error sensitivity.

AuthorsOpenReview ↗

W2 / Q5 Baseline fidelity and controlled comparison && W3 / Q4 Small cross-block gain

W2 / Q5 Baseline fidelity and controlled comparison

Because no author-released code could be located at evaluation time, we built a paper-based reference. It retains the KVTC paper's PCA, adaptive mixed-bit quantization and nvCOMP DEFLATE entropy coding. The key positional-encoding difference is that KVTC compresses pre-RoPE keys, whereas our original evaluation compresses native KV after RoPE. Our layout adaptation uses joint PCA across four heads within a layer rather than across layers. These disclosed adaptations preserve the core compression pipeline.

The GLM-5.3-Flash results in our paper already evaluate a RoPE-free MLA latent representation. On this model, AlignPred achieves lower storage than our KVTC reference at comparable reconstruction error, demonstrating that its advantage also holds for the evaluated RoPE-free cache representation.

For KVCodec, we use software HEVC with the paper’s lossless=1 setting. Video coding introduces no additional reconstruction error; the reported timing corresponds to this software backend.

W3 / Q4 Small cross-block gain

AlignPred delivers substantial overall storage savings as a complete method for long-term KV-cache management. Table 2 isolates the smaller but valuable incremental contribution of alignment-enabled cross-block prediction. Even a few percent of additional savings can translate into meaningful capacity and cost reductions for large-scale persistent KV storage.

The correlations within KV caches differ across models. DeepSeek’s native CSA/C4 compression may leave less cross-layer redundancy to exploit. Appendix Figure 12 supports this explanation: aligned inter-layer same-bit mutual information is weaker than in GLM.

We further toggled intra-block prediction on five DeepSeek documents. “Fraction retained” measures how much cross-block saving remains when intra-block prediction is enabled, relative to when it is disabled.

DeepSeek intra-block and cross-block prediction ablation

Error budget εcontrolCross-block saving without intra-block prediction bpvCross-block saving with intra-block prediction bpvFraction retainedReduction from the intra-block-only control
0.0255360.0953140.07993683.87%1.54%
0.1021430.0879700.07624386.67%2.39%
0.2042870.0731070.06987695.58%3.11%

On DeepSeek, 84%–96% of the cross-block gain remains when intra-block prediction is enabled, showing that intra-block prediction does not account for most of this benefit. As the error budget increases, the retained fraction rises from 83.87% to 95.58%, while the additional packet reduction over the intra-block-only control rises from 1.54% to 3.11%.

AuthorsOpenReview ↗

W4 / Q3 Fixed alignment and domain transfer

W4 / Q3 Fixed alignment and domain transfer

Our rotation acts on each token’s vector within each KV head. For Qwen3.5-9B, each matrix is 256 × 256 and is shared across token positions and inputs.

On ten held-out documents, five legal and five biomedical, fixed GPA captures 80.26% of the alignment-error reduction achieved by document-fitted pairwise maps, relative to no alignment.

We also compared GPA with pairwise maps fitted on the same independent calibration data, keeping the remaining codec unchanged. Total storage includes packets, indices and parameters shared across the ten documents.

Fixed GPA versus calibration-fitted pairwise maps

Error budget εcontrolFrozen GPA packet bpvCalibration-pairwise packet bpvFrozen GPA total bpvCalibration-pairwise total bpv
0.0266814.53604.51494.56144.5778
0.1067242.56452.54502.58992.6079

Pairwise maps slightly reduce packet size, but GPA uses fewer parameters and achieves lower total storage at both budgets.

Without refitting, the frozen alignment parameters also retain useful cross-block savings in both new domains:

Qwen 32K cross-block savings with frozen alignment parameters Domain Error budget εcontrol Alignment + intra-block packet bpv Full AlignPred packet bpv Additional saving Legal 0.026681 4.7046 4.5365 3.57% Legal 0.106724 2.7244 2.5648 5.86% Biomedical 0.026681 4.6963 4.5341 3.45% Biomedical 0.106724 2.7169 2.5627 5.67%

These results support fixed GPA as a compact, reusable mapping, with 3.45%–5.86% additional packet savings on the tested new domains.

Reviewer_WEdwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes AlignPred, an error-bounded compression codec for persistent KV caches. Targeting the storage requirements of retaining KV caches across turns and sessions in long-lived agent conversations, AlignPred maps KV blocks from different attention heads and layers into a common coordinate system via local orthogonal coordinate alignment. It then combines a two-stage predictive coding scheme to achieve KV cache compression within a configurable reconstruction error budget.

Critical strengths

(1) The method takes practical engineering deployment into account, adopting a pipelined CPU–GPU cooperation approach to improve computational efficiency.

(2) The experiments are relatively comprehensive, featuring detailed comparisons against multiple baselines and covering several mainstream attention architectures, including GQA, MLA, and CSA, which demonstrates the strong generalizability of AlignPred.

Critical weaknesses

(1) The authors only evaluate their method on LongBench-Pro, which covers a limited set of task scenarios. Given the current landscape of LLM development and the authors' use of frontier models such as GLM-5.3-Flash and DeepSeek-V4-Flash, it is necessary to evaluate the method on extreme agent scenarios, such as complex reasoning and long-horizon tool-use chains, to substantiate its generalizability and robustness.

(2)When grouping attention heads, the authors rely entirely on the model's native adjacent structural grouping, which is not necessarily optimal for compression gains. Although the grouping sensitivity analysis in Appendix C.7 indicates that larger or better grouping configurations can further improve the compression ratio, the authors do not explore this direction further, leaving this aspect of the work feeling somewhat incomplete. Since this is a fairly natural extension, the authors should provide a more thorough exploratory evaluation and conclusive analysis regarding grouping strategies.

Questions for authors

The authors only evaluate the efficiency of the CPU-based encoding and decoding process without comparing it against native GPU encoding/decoding. Native GPU decoding can eliminate the CPU-to-GPU data transfer overhead. Have the authors considered this trade-off?

2 comments
AuthorsOpenReview ↗

W1 Agent workloads beyond LongBench Pro && W2 Exploration of grouping strategies

W1 Agent workloads beyond LongBench Pro

We extended the evaluation to adaptive multi-turn tool use with 40 BFCL cases per model: 20 base and 20 long-context cases. The table reports BFCL-checker passes / total cases at the FP8-reference error budget ε and at 4ε; higher is better.

BFCL multi-turn tool-use success counts

ModelSubsetNativeFP8AlignPred (εref)AlignPred (4εref)
Qwen3.5-9BBase12/2014/2013/2013/20
Qwen3.5-9BLong-context8/208/208/2010/20
Qwen3.5-9BOverall20/4022/4021/4023/40
DeepSeek-V4-FlashBase11/2010/2010/2010/20
DeepSeek-V4-FlashLong-context13/2013/2013/2012/20
DeepSeek-V4-FlashOverall24/4023/4023/4022/40
GLM-5.3-FlashBase11/2011/2013/2011/20
GLM-5.3-FlashLong-context9/209/209/209/20
GLM-5.3-FlashOverall20/4020/4022/4020/40

Across the three models, aggregate task success is broadly comparable to Native. This extends the evidence from long-context question answering to adaptive multi-turn tool use.

W2 Exploration of grouping strategies

Response

We compared native adjacency, random grouping and similarity-based grouping on five held-out documents per model at the FP8-reference error budget. Native adjacency remains competitive; the random result is the best of three fixed seeds.

Storage comparison across grouping strategies

Fixed-size groupingQwen bpvGLM bpvDeepSeek bpv
Native adjacency4.35265.02555.0445
Random grouping (minimum over 3 seeds)4.64455.08815.0997
Similarity grouping4.38765.06285.0102

The default grouping offers a simple, intuitive choice with competitive compression.

Q1 CPU and GPU codec tradeoff

We added a GPU implementation using nvCOMP ANS and compared it with the CPU compact-rANS path under concurrent inference. Both retain the same alignment, prediction and error-budget settings.

we allocate 32 CPU cores or 8 SMs on the inference GPU to 32K background decoding and compare decoding time (excluding transfers) and foreground P95 TTFT.

Concurrent foreground inference and background decoding (ms; lower is better)

ModelBackground decode CPUBackground decode GPUForeground TTFT CPUForeground TTFT GPU
Qwen3.5-9B208.9506.1215.5248.3
DeepSeek-V4-Flash47.886.31161.01104.3
GLM-5.3-Flash84.5267.61139.31070.3

Under this resource allocation, CPU decoding takes 44.7%–68.4% less time than GPU decoding across the three models.

Our CPU-worker design is motivated by a practical deployment requirement: application teams are often reluctant to run KV-cache encoding and decoding on inference GPUs. Codec operations, particularly entropy coding, make limited use of GPU matrix acceleration yet compete with inference for GPU resources. We therefore place most codec computation on CPU workers to preserve GPU capacity for serving requests.

AuthorsOpenReview ↗

Thank you for your thoughtful feedback.

Thank you for your thoughtful feedback. We have added agent-workload evaluations, grouping analyses and CPU/GPU comparisons to address your concerns. Please feel free to ask any follow-up questions.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your constructive feedback. We briefly summarize our responses and revisions:

  • WEdw: We extended the evaluation to multi-turn tool use, including 40 GLM Agent cases, and added CPU/GPU decoding comparisons measuring both background decoding time and foreground latency, together with temporary GPU-memory measurements.

  • phr5: We have clarified error control, baseline adaptations, alignment contributions and asynchronous restoration, supported by additional experiments.

  • Wda2: We clarified the deployment motivation and methodological distinction, emphasized the complete error-controlled compression pipeline, and added an explicit component breakdown.

The revised PDF incorporates these changes, highlighted in blue, with supplementary experiments in Appendix F. We welcome further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.