- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Latent Compaction, which trains a reusable compact KV cache per document with frozen model weights. Initialization fills a fixed logical budget with three complementary entry types: high-scoring KVZip+ preservers, high-attention anchors, and farthest-first diversifiers. Training first fits attention outputs and log-normalizers using teacher queries, then performs end-to-end refinement via top-20-plus-tail distillation and an LM loss using teacher rollouts from generic prompts. At inference, the compressed document cache is combined with ordinary KV entries of subsequent question/generated tokens through log-normalizer weighting.
Critical strengths
-
Novel and well-composed method design. It combines KV selection with learned compression; the three initialization types target reconstruction importance, attention importance, and key-space coverage, while Stage 1 and Stage 2 are complementary and the normalizer-aware combination has a clear mathematical form.
-
No downstream questions required, fitting document-cache reuse. Document-specific supervision uses only teacher rollouts from fixed generic prompts, avoiding information leakage. The same document cache is reused across multiple questions, e.g., 20 questions per document in Phonebook Lookup.
-
Broad experimental coverage and thorough ablations. Evaluation spans three benchmarks, multiple model scales, and two model families, with systematic ablations on initialization, budget allocation, two-stage training, log-Z loss, attention bias, normalizer weighting, and training duration, supporting key design choices.
Critical weaknesses
-
High per-document optimization cost. Each document requires teacher rollouts for 480 generic prompts, 1,000 updates per layer in Stage 1, and 1,000 updates in Stage 2. For large or dynamic document collections, preparation overhead may exceed benefits; the paper does not quantify the cost–benefit trade-off.
-
Only logical KV-pair budgets are reported; actual memory and inference speed are not measured. Deployment is also affected by precision, biases, padding, metadata, and the custom attention implementation. The compressed path does not use FlashAttention/fused SDPA, so actual memory savings and latency benefits remain unclear.
-
Performance degrades sharply under extreme compression, and the method is not uniformly superior. Phonebook Lookup approaches zero at 64× and beyond, LongBench scores still decline with compression, and text summarization can outperform on RULER at the most aggressive ratios. The applicable boundary needs clearer characterization.
-
Evaluation scope and baseline fairness could be strengthened. Only three benchmarks are covered, and Phonebook Lookup is a synthetic retrieval task. SnapKV and PyramidKV are adapted to query-agnostic versions, Attention Matching uses a specific variant, and text-summary budget alignment is not fully equivalent to KV compression, requiring clearer fairness discussion.
-
High method complexity and system integration cost. A custom attention path is needed to obtain attention outputs, log-normalizers, per-entry biases, and ragged masks simultaneously, and keys, values, biases, and masks must be stored. This requires substantial inference-stack changes, and code is only “planned to be released,” so full reproduction still depends on considerable engineering effort.
Questions for authors
Does Latent Compaction actually bring real benefits in real-world deployments?
2 comments
Rebuttal part 2/2
16K baseline comparison
Both tables use Qwen3-1.7B at 16K. The RULER table uses the Figure 1 reference scores, and the Phonebook table uses the updated Latent Compaction results. Both Cartridges runs use the same KV budget as Latent Compaction at each ratio. The larger run uses 68× the training data and iterations of the initial run and roughly matches the official implementation's training scale. The second Attention Matching run uses the same 480 teacher prompts as Latent Compaction.
RULER, 13-task macro score (%)
| Method | 2x | 4x | 8x | 16x | 32x | 64x |
|---|---|---|---|---|---|---|
| Latent Compaction | 83.30 | 80.27 | 75.21 | 70.97 | 62.49 | 53.40 |
| KVZip+ | 83.28 | 78.54 | 57.74 | 41.46 | 25.76 | 16.64 |
| Attention Matching | 69.72 | 49.07 | 27.40 | 21.94 | 20.67 | 13.92 |
| Cartridges | 62.28 | 39.72 | 27.37 | 17.59 | 13.67 | 11.34 |
| Cartridges (68× training scale) | 78.30 | 73.29 | 62.57 | 56.47 | 38.47 | 28.57 |
| Attention Matching with same teacher prompts | 54.08 | 31.73 | 18.21 | 18.75 | 17.57 | 16.57 |
Phonebook, exact recall (%)
| Method | 2x | 4x | 8x | 16x | 32x | 64x |
|---|---|---|---|---|---|---|
| Latent Compaction | 89.50 | 81.50 | 74.50 | 70.50 | 56.75 | 5.50 |
| KVZip+ | 90.75 | 78.50 | 11.00 | 2.00 | 0.25 | 0.00 |
| Attention Matching | 60.75 | 2.75 | 0.00 | 0.00 | 0.00 | 0.00 |
| Cartridges | 45.50 | 11.75 | 3.25 | 1.00 | 0.25 | 0.00 |
| Cartridges (68× training scale) | 67.25 | 40.50 | 23.75 | 19.50 | 7.25 | 3.00 |
| Attention Matching with same teacher prompts | 47.25 | 0.25 | 0.50 | 0.00 | 0.00 | 0.00 |
A800 measurements and derived quantities
Qwen3-4B, 16K input, 16× logical document KV compaction, one 80 GB NVIDIA A800. Pairs are Latent Compaction with our custom fused kernel / reused full KV with FlashAttention. Throughput uses 128 output tokens per request. Matched-batch speedup divides full-KV batch time by compact-cache batch time.
| Batch size | Memory (GiB) | Time per batch (s) | Throughput (tokens/s) | Matched-batch speedup | Break-even queries |
|---|---|---|---|---|---|
| 1 | 15.7 / 17.8 | 2.9 / 9.5 | 44.1 / 13.5 | 3.28× | 246 |
| 2 | 15.8 / 20.0 | 4.1 / 14.4 | 62.4 / 17.8 | 3.51× | 316 |
| 4 | 16.1 / 24.6 | 6.8 / 23.0 | 75.3 / 22.3 | 3.38× | 404 |
| 8 | 16.8 / 33.7 | 9.2 / 30.9 | 111.3 / 33.1 | 3.36× | 600 |
| 16 | 18.1 / 51.8 | 15.6 / 52.0 | 131.3 / 39.4 | 3.33× | 720 |
| 32 | 22.6 / OOM | 24.4 / OOM | 167.9 / OOM | — | — |
| 64 | 31.8 / OOM | 36.2 / OOM | 226.3 / OOM | — | — |
| 128 | 46.0 / OOM | 48.8 / OOM | 335.7 / OOM | — | — |
| 256 | 60.6 / OOM | 74.4 / OOM | 440.4 / OOM | — | — |
For matched batches of size , the first strictly lower cumulative time occurs after queries. We charge 1,620 s of preparation to compaction and start the baseline with its full KV cache already resident. The highest-throughput tested configurations average 0.291 and 3.250 s/query, yielding an estimated crossover at 547.4 queries by rate and about 550 queries when combining the measured batch sizes. At 768 queries, three compact batches of 256 take 30.72 min including preparation, and 48 full-KV batches of 16 take 41.60 min. These calculations use the recorded batch durations and exclude waiting for requests to accumulate.
Break-even against KVZip+
Qwen3-1.7B at 16K context on the 13-task RULER macro average. Latent Compaction at 32× scores 62.49 and KVZip+ at 8× scores 57.74. Per document, preparation takes 1,404 s for Latent Compaction and 2.3 s for KVZip+. At batch size 1, serving takes 1.9 s versus 15.2 s per query, an 8.0× serving speedup.
After queries, cumulative time is s for Latent Compaction and s for KVZip+. The curves cross at queries. The first whole-query count where Latent Compaction is faster is 106 queries per document. At that point, cumulative times are 1,605.4 s and 1,613.5 s, respectively.
Rebuttal part 1/2
Dear Reviewer fnXE,
Thank you for the time and effort you put into reviewing our paper and for asking us to measure the costs of deployment.
1. Per-document optimization cost
On one 80 GB NVIDIA A800, preparing a 16K Qwen3-4B document at 16× took 27.0 min (11.2 min for teacher QA generation and replay, 15.8 min for fitting). Against reused full KV, compaction pays back preparation at 246 serial queries, 720 queries when both methods use batch 16, or about 550 queries at each method's highest-throughput measured batch size. Candidate workloads include support queries about a troubleshooting manual and batched audits of a policy handbook. The savings depend on how many queries each version receives.
On 16K RULER with Qwen3-1.7B, Latent Compaction at 32× scores 62.49, versus 57.74 for KVZip+ at 8×. Serving at batch size 1 takes 1.9 vs. 15.2 s per query. Including 23.4 min vs. 2.3 s of preparation, Latent Compaction first becomes faster at 106 queries per document. See the cumulative timing calculation.
Our proof of concept demonstrates that a cache fitted at test time can be reused across queries. Batched teacher generation, batched replay, and shorter fitting schedules could reduce preparation time. We leave these optimizations to future work.
2. Physical memory and inference speed
Using our custom fused kernel on the A800, Latent Compaction takes 15.6 s at batch 16 versus 52.0 s for reused full KV with FlashAttention, a 3.33× speedup. GPU memory is 18.1 vs. 51.8 GiB. In this benchmark, full KV runs out of memory at batch 32, while compaction fits at batch 256 in 60.6 GiB and reaches 11.18× the throughput of the fastest measured full-KV batch. Serving backends that share prefix storage across requests can change the maximum batch sizes. See the A800 measurements.
3. Degradation at extreme compression
At 16K, Phonebook recall drops from 88.75% to 2.25% on Qwen3-4B and from 90.50% to 9.00% on Qwen3-8B when compression increases from 16× to 32×. On the 16K RULER suite, Qwen3-4B compaction at 32× scores 66.43, close to KVZip+ at 16× (66.17). The usable compression ratio depends on the task. We will report these comparisons alongside the serving measurements.
4. Evaluation scope and baseline fairness
The Qwen3-1.7B LongBench comparison uses the same 320 examples for all six methods. Latent Compaction leads at 16×, KVZip+ at 4× and 8×, and adapted SnapKV at 32×. On 16K RULER and Phonebook, we added two Cartridges runs at the same KV budget. The larger run roughly matches the official implementation's training scale. We also evaluated Attention Matching with the same 480 teacher prompts. We will label the query-agnostic SnapKV and PyramidKV adaptations and report actual summary lengths alongside the logical KV budgets. See the 16K baseline comparison.
5. Integration and reproducibility
We have implemented a fused attention kernel for compact-cache inference. The inference path supports per-entry biases and masks for padded document slots, and uses a shared normalizer for the compacted document and subsequent tokens. The appendix documents the attention calculation and cache layout. We will release all source code and model checkpoints as soon as the review process ends.
Thank you again for raising these deployment questions. We’d be happy to discuss the measurements or the implementation in more detail. If the new evidence addresses your concerns, please consider raising your score.
Figure 5 allocation sweep
Phonebook recall (%) at 16× compression. The left panel varies the fraction of document KV slots assigned to preservers. The right panel varies the fraction of the remaining slots assigned to anchors. Each mean gives equal weight to 4K, 8K, and 16K.