acceptodds
Under review as a conference paper at ICLR 2027

Latent Compaction with Test-Time Training

Abstract

Long contexts enlarge key-value (KV) caches, increasing memory use and attention costs during generation. Compressing these caches must preserve information for unseen questions. We introduce Latent Compaction, which learns a compact document-specific cache through test-time training. We initialize cache entries using reconstruction importance, teacher attention, and key-space diversity. Training first fits attention outputs and log-normalizers to those of the full cache, with reconstruction-selected entries held fixed. It then jointly optimizes the entire cache through end-to-end distillation from the full-cache model. Model weights remain frozen throughout, and training uses only generic prompts about the document, without access to downstream questions or answers. Experiments on RULER, LongBench, and Phonebook Lookup demonstrate substantial gains over budget-matched compression baselines. On a RULER, for example, our method achieves 96.7% accuracy at compression, compared with 40.0% for KVZip+. The method also transfers across model scales and families, with retrieval gains on both Qwen3 and MiniCPM5. These results show that treating the KV cache as a neural network and optimizing it with test-time training can preserve retrieval accuracy under aggressive compression.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV compression, test-time training, long-context

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fnXEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Latent Compaction, which trains a reusable compact KV cache per document with frozen model weights. Initialization fills a fixed logical budget with three complementary entry types: high-scoring KVZip+ preservers, high-attention anchors, and farthest-first diversifiers. Training first fits attention outputs and log-normalizers using teacher queries, then performs end-to-end refinement via top-20-plus-tail distillation and an LM loss using teacher rollouts from generic prompts. At inference, the compressed document cache is combined with ordinary KV entries of subsequent question/generated tokens through log-normalizer weighting.

Critical strengths
  1. Novel and well-composed method design. It combines KV selection with learned compression; the three initialization types target reconstruction importance, attention importance, and key-space coverage, while Stage 1 and Stage 2 are complementary and the normalizer-aware combination has a clear mathematical form.

  2. No downstream questions required, fitting document-cache reuse. Document-specific supervision uses only teacher rollouts from fixed generic prompts, avoiding information leakage. The same document cache is reused across multiple questions, e.g., 20 questions per document in Phonebook Lookup.

  3. Broad experimental coverage and thorough ablations. Evaluation spans three benchmarks, multiple model scales, and two model families, with systematic ablations on initialization, budget allocation, two-stage training, log-Z loss, attention bias, normalizer weighting, and training duration, supporting key design choices.

Critical weaknesses
  1. High per-document optimization cost. Each document requires teacher rollouts for 480 generic prompts, 1,000 updates per layer in Stage 1, and 1,000 updates in Stage 2. For large or dynamic document collections, preparation overhead may exceed benefits; the paper does not quantify the cost–benefit trade-off.

  2. Only logical KV-pair budgets are reported; actual memory and inference speed are not measured. Deployment is also affected by precision, biases, padding, metadata, and the custom attention implementation. The compressed path does not use FlashAttention/fused SDPA, so actual memory savings and latency benefits remain unclear.

  3. Performance degrades sharply under extreme compression, and the method is not uniformly superior. Phonebook Lookup approaches zero at 64× and beyond, LongBench scores still decline with compression, and text summarization can outperform on RULER at the most aggressive ratios. The applicable boundary needs clearer characterization.

  4. Evaluation scope and baseline fairness could be strengthened. Only three benchmarks are covered, and Phonebook Lookup is a synthetic retrieval task. SnapKV and PyramidKV are adapted to query-agnostic versions, Attention Matching uses a specific variant, and text-summary budget alignment is not fully equivalent to KV compression, requiring clearer fairness discussion.

  5. High method complexity and system integration cost. A custom attention path is needed to obtain attention outputs, log-normalizers, per-entry biases, and ragged masks simultaneously, and keys, values, biases, and masks must be stored. This requires substantial inference-stack changes, and code is only “planned to be released,” so full reproduction still depends on considerable engineering effort.

Questions for authors

Does Latent Compaction actually bring real benefits in real-world deployments?

2 comments
AuthorsOpenReview ↗

Rebuttal part 1/2

Dear Reviewer fnXE,

Thank you for the time and effort you put into reviewing our paper and for asking us to measure the costs of deployment.

1. Per-document optimization cost

On one 80 GB NVIDIA A800, preparing a 16K Qwen3-4B document at 16× took 27.0 min (11.2 min for teacher QA generation and replay, 15.8 min for fitting). Against reused full KV, compaction pays back preparation at 246 serial queries, 720 queries when both methods use batch 16, or about 550 queries at each method's highest-throughput measured batch size. Candidate workloads include support queries about a troubleshooting manual and batched audits of a policy handbook. The savings depend on how many queries each version receives.

On 16K RULER with Qwen3-1.7B, Latent Compaction at 32× scores 62.49, versus 57.74 for KVZip+ at 8×. Serving at batch size 1 takes 1.9 vs. 15.2 s per query. Including 23.4 min vs. 2.3 s of preparation, Latent Compaction first becomes faster at 106 queries per document. See the cumulative timing calculation.

Our proof of concept demonstrates that a cache fitted at test time can be reused across queries. Batched teacher generation, batched replay, and shorter fitting schedules could reduce preparation time. We leave these optimizations to future work.

2. Physical memory and inference speed

Using our custom fused kernel on the A800, Latent Compaction takes 15.6 s at batch 16 versus 52.0 s for reused full KV with FlashAttention, a 3.33× speedup. GPU memory is 18.1 vs. 51.8 GiB. In this benchmark, full KV runs out of memory at batch 32, while compaction fits at batch 256 in 60.6 GiB and reaches 11.18× the throughput of the fastest measured full-KV batch. Serving backends that share prefix storage across requests can change the maximum batch sizes. See the A800 measurements.

3. Degradation at extreme compression

At 16K, Phonebook recall drops from 88.75% to 2.25% on Qwen3-4B and from 90.50% to 9.00% on Qwen3-8B when compression increases from 16× to 32×. On the 16K RULER suite, Qwen3-4B compaction at 32× scores 66.43, close to KVZip+ at 16× (66.17). The usable compression ratio depends on the task. We will report these comparisons alongside the serving measurements.

4. Evaluation scope and baseline fairness

The Qwen3-1.7B LongBench comparison uses the same 320 examples for all six methods. Latent Compaction leads at 16×, KVZip+ at 4× and 8×, and adapted SnapKV at 32×. On 16K RULER and Phonebook, we added two Cartridges runs at the same KV budget. The larger run roughly matches the official implementation's training scale. We also evaluated Attention Matching with the same 480 teacher prompts. We will label the query-agnostic SnapKV and PyramidKV adaptations and report actual summary lengths alongside the logical KV budgets. See the 16K baseline comparison.

5. Integration and reproducibility

We have implemented a fused attention kernel for compact-cache inference. The inference path supports per-entry biases and masks for padded document slots, and uses a shared normalizer for the compacted document and subsequent tokens. The appendix documents the attention calculation and cache layout. We will release all source code and model checkpoints as soon as the review process ends.

Thank you again for raising these deployment questions. We’d be happy to discuss the measurements or the implementation in more detail. If the new evidence addresses your concerns, please consider raising your score.

Figure 5 allocation sweep

Phonebook recall (%) at 16× compression. The left panel varies the fraction of document KV slots assigned to preservers. The right panel varies the fraction of the remaining slots assigned to anchors. Each mean gives equal weight to 4K, 8K, and 16K.

FractionPreservers 4KPreservers 8KPreservers 16KPreservers meanAnchors 4KAnchors 8KAnchors 16KAnchors mean
0%0.750.500.250.5066.5049.5048.2554.75
10%7.753.503.254.8371.5075.5064.2570.42
25%36.2541.7522.5033.5087.7583.7570.5080.67
50%81.2568.5064.2571.3382.5069.7569.2573.83
75%87.7583.7570.5080.6775.7566.5062.2568.17
90%61.2564.0049.0058.0874.5066.2562.2567.67
100%12.005.502.006.5071.5061.7557.7563.67
AuthorsOpenReview ↗

Rebuttal part 2/2

16K baseline comparison

Both tables use Qwen3-1.7B at 16K. The RULER table uses the Figure 1 reference scores, and the Phonebook table uses the updated Latent Compaction results. Both Cartridges runs use the same KV budget as Latent Compaction at each ratio. The larger run uses 68× the training data and iterations of the initial run and roughly matches the official implementation's training scale. The second Attention Matching run uses the same 480 teacher prompts as Latent Compaction.

RULER, 13-task macro score (%)

Method2x4x8x16x32x64x
Latent Compaction83.3080.2775.2170.9762.4953.40
KVZip+83.2878.5457.7441.4625.7616.64
Attention Matching69.7249.0727.4021.9420.6713.92
Cartridges62.2839.7227.3717.5913.6711.34
Cartridges (68× training scale)78.3073.2962.5756.4738.4728.57
Attention Matching with same teacher prompts54.0831.7318.2118.7517.5716.57

Phonebook, exact recall (%)

Method2x4x8x16x32x64x
Latent Compaction89.5081.5074.5070.5056.755.50
KVZip+90.7578.5011.002.000.250.00
Attention Matching60.752.750.000.000.000.00
Cartridges45.5011.753.251.000.250.00
Cartridges (68× training scale)67.2540.5023.7519.507.253.00
Attention Matching with same teacher prompts47.250.250.500.000.000.00

A800 measurements and derived quantities

Qwen3-4B, 16K input, 16× logical document KV compaction, one 80 GB NVIDIA A800. Pairs are Latent Compaction with our custom fused kernel / reused full KV with FlashAttention. Throughput uses 128 output tokens per request. Matched-batch speedup divides full-KV batch time by compact-cache batch time.

Batch sizeMemory (GiB)Time per batch (s)Throughput (tokens/s)Matched-batch speedupBreak-even queries
115.7 / 17.82.9 / 9.544.1 / 13.53.28×246
215.8 / 20.04.1 / 14.462.4 / 17.83.51×316
416.1 / 24.66.8 / 23.075.3 / 22.33.38×404
816.8 / 33.79.2 / 30.9111.3 / 33.13.36×600
1618.1 / 51.815.6 / 52.0131.3 / 39.43.33×720
3222.6 / OOM24.4 / OOM167.9 / OOM——
6431.8 / OOM36.2 / OOM226.3 / OOM——
12846.0 / OOM48.8 / OOM335.7 / OOM——
25660.6 / OOM74.4 / OOM440.4 / OOM——

For matched batches of size BB, the first strictly lower cumulative time occurs after B(⌊1620/(tfull−tcompact)⌋+1)B(\lfloor 1620/(t_{\mathrm{full}}-t_{\mathrm{compact}})\rfloor+1) queries. We charge 1,620 s of preparation to compaction and start the baseline with its full KV cache already resident. The highest-throughput tested configurations average 0.291 and 3.250 s/query, yielding an estimated crossover at 547.4 queries by rate and about 550 queries when combining the measured batch sizes. At 768 queries, three compact batches of 256 take 30.72 min including preparation, and 48 full-KV batches of 16 take 41.60 min. These calculations use the recorded batch durations and exclude waiting for requests to accumulate.

Break-even against KVZip+

Qwen3-1.7B at 16K context on the 13-task RULER macro average. Latent Compaction at 32× scores 62.49 and KVZip+ at 8× scores 57.74. Per document, preparation takes 1,404 s for Latent Compaction and 2.3 s for KVZip+. At batch size 1, serving takes 1.9 s versus 15.2 s per query, an 8.0× serving speedup.

After nn queries, cumulative time is 1,404+1.9n1{,}404 + 1.9n s for Latent Compaction and 2.3+15.2n2.3 + 15.2n s for KVZip+. The curves cross at (1,404−2.3)/(15.2−1.9)=105.39(1{,}404-2.3)/(15.2-1.9)=105.39 queries. The first whole-query count where Latent Compaction is faster is 106 queries per document. At that point, cumulative times are 1,605.4 s and 1,613.5 s, respectively.

Reviewer_AsfdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces latent compaction, which initializes a compact cache with three types of tokens: preservers, anchors, and diversifiers. The training procedure consists of two stages. In the first stage, only the anchors and diversifiers are updated by matching attention distributions and log-normalizers, following an approach similar to attention matching. In the second stage, the entire compact cache is jointly optimized through end-to-end distillation from the full-cache model, using teacher rollouts generated with generic prompts. The authors demonstrate strong performance across various benchmarks (RULER, LongBench, and Phonebook), spanning multiple model scales, model families, and compression ratios.

Critical strengths
  1. The problem is timely and important. Although the proposed method is relatively simple and combines existing techniques, it demonstrates strong empirical effectiveness.

  2. The paper is generally well-written and easy to follow.

  3. The ablation studies are thorough, covering initialization strategies, both training stage-1/2, and architectural variations. The authors also include analyses across different training durations.

  4. The proposed method demonstrates consistent effectiveness across various benchmarks.

Critical weaknesses
  1. Practical Motivation and Efficiency Trade-offs I am not fully convinced that training a separate KV cache for each document is a practical scenario. Several additional analyses would need to claim the practical value of the proposed approach.
  • Practicality of repeated queries over the same document. The paper should provide at least a brief empirical analysis or real-world motivation demonstrating how common it is to process a document once and then issue multiple consecutive queries about it. This is important for justifying the per-document training setup.
  • Break-even point for compression efficiency. The proposed method incurs an upfront training cost for each document, in addition to the cost of KV cache compression. Depending on the number of subsequent queries, there should be a break-even point beyond which the reduced cost outweighs this preparation overhead. The authors should quantify this trade-off by comparing accuracy and cumulative computational cost or wall-clock time as a function of the number of queries, ideally against existing baselines. Such an analysis would clarify when the proposed method becomes more efficient in practice.
  • Positioning relative to RestoreKV. Although RestoreKV may be a contemporary work, a more detailed discussion of its differences from the proposed approach and the respective practical advantages would help clarify the positioning of this paper, even if a direct experimental comparison is not appropriate.
  1. Novelty claims and Insufficient Ablation of the Proposed Components The proposed method seems to combine existing techniques, but it remains unclear which aspects constitute the key novel contributions and what distinguishes this combination from prior approaches. While I believe novelty does not necessarily require, the paper should more clearly describe and experimentally validate the distinctive design choices. In particular, Stage 1 seems similar to attention matching, while Stage 2 resembles Cartridges. The paper should more carefully compare these components with the corresponding mechanisms in prior works and show these differences in the ablation studies.
  • Stage 1: Comparison with attention matching. Attention matching should serve as a direct baseline, with additional ablations isolating the effects of the proposed modifications. For example: 1) Closed-form vs gradient-based, 2) Fixed versus learnable keys, including comparisons.
  • Stage 2: Comparison with Cartridges. It would be great if the authors include ablations that isolate the specific modifications introduced relative to the original Cartridges approach, rather than evaluating only the complete proposed pipeline. This would help determine whether the proposed changes provide meaningful gains over the existing method.
  • Baseline coverage. Given that the similarity of the method, I believe Cartridges should be included as a baseline across the experiments. In addition, it would be helpful to clarify whether the reported attention-matching results are reproduced using the original method as described in its paper.
Questions for authors

Wrote above

2 comments

Dear Reviewer Asfd,

Thank you for taking the time to review our paper and for asking where the method is useful and how it compares with prior work.

1. Practical motivation, break-even, and RestoreKV

Break-even against full KV. On one 80 GB A800, preparing a 16K Qwen3-4B document at 16× took 27.0 min (11.2 min for teacher QA generation and replay, 15.8 min for fitting). The compact cache takes 2.9 s per serial query versus 9.5 s with reused full KV. That pays back preparation at 246 serial queries. At each method's highest-throughput measured batch size, the crossover is about 550 queries (batch 256 for compaction, 16 for full KV).

Break-even against KVZip+. On 16K RULER with Qwen3-1.7B, Latent Compaction at 32× scores 62.49 and KVZip+ at 8× scores 57.74. Latent Compaction uses one quarter of the logical document KV slots and saves 13.3 s per query at batch size 1 (1.9 s vs. 15.2 s). Including 23.4 min of preparation versus 2.3 s for KVZip+, Latent Compaction first becomes faster at 106 queries per document. See the cumulative timing calculation.

When reuse pays off. A support team could prepare a troubleshooting manual once per product release and reuse it across tickets. An offline audit could batch hundreds of checklist questions against one edition of a policy handbook. In both cases, the queries per document version determine whether preparation pays off.

The paper is a proof of concept for fitting a reusable document cache at test time, before downstream questions are known. Batching teacher generation and replay could reduce rollout costs, and shorter fitting schedules could reduce optimization time. Improving preparation speed is future work.

RestoreKV. RestoreKV uses a shared trained adapter to construct a cache in one adapted pass after prefill. Latent Compaction keeps the model weights frozen and spends compute fitting the cache to each document. RestoreKV

2. Contribution and ablations against Attention Matching and Cartridges

Stage 1. The solid curve in Figure 6 (left) holds Stage 2 at zero. The score changes from 10.50 at initialization to 57.50 after 1,000 Stage 1 updates and 58.00 after 2,000. This isolates the gain from attention fitting before distillation.

The revision clarifies how Stage 1 builds on Attention Matching and how Stage 2 builds on Cartridges' self-study distillation.

Baseline comparisons. At 16×, Attention Matching with the same 480 prompts scores 18.75 on RULER and 0.00% on Phonebook. At equal KV budgets, Cartridges reaches 56.47 and 19.50% with 68× the training data and iterations of our initial run. Latent Compaction scores 70.97 and 70.50%. The full sweeps cover six compression ratios for both Cartridges runs and both Attention Matching configurations. Isolating the solver and key trainability requires holding the allocation and initialization fixed as well.

Thanks again for your feedback. We’d welcome further discussion about the break-even results or the comparison with prior work. If our response addresses your concerns, we hope you’ll consider raising your score.

Thank you for your responses and explanations. I have carefully read the other reviewers' comments and responses, as well as the response to my own review. However, I believe my main concerns remain unresolved.

  1. Practicality and Computational Cost: The preparation cost remains substantially higher than that of existing methods (e.g., KV-zip+), and the method seems to become beneficial over full KV caching only after hundreds of queries. Even if the method has value as a proof of concept, I believe the cost seems disproportionately high relative to the benefits. I believe a more concrete discussion of how to reduce this overhead is necessary, ideally bringing it closer to existing methods. If I have misunderstood the results, a clearer cost-benefit comparison in a table or figure would be helpful.

  2. Regarding the contributions, I still find it difficult to clearly distinguish the proposed method from existing approaches based on the current responses. In particular, I would appreciate a more explicit comparison with prior methods that clarifies the key methodological differences and the specific contributions of this work.

I understand that time is limited, but I would be willing to reconsider my assessment if further evidence or clarification is provided. For now, my main concerns remain unresolved.

Reviewer_4idXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compresses a long document's KV cache before any questions are known, so the small cache can be reused for many questions. The compressed cache is treated as trainable parameters and optimized for each document, with the model weights frozen. It is filled with KVZip+-selected tokens plus anchor and diversity entries, then trained in two stages: matching attention outputs layer by layer, then distilling from the full-cache model on its answers to 480 generic prompts. On RULER, LongBench, and Phonebook Lookup (mainly Qwen3-1.7B), it beats selection-based baselines, most clearly on Phonebook: about 57% vs. near 0 at 16K context and 16× compression (Fig 1(d)).

Critical strengths
  1. A large and clear gain on synthetic retrieval under strong compression. On Phonebook Lookup with Qwen3-1.7B, KVZip+, Attention Matching, SnapKV, and PyramidKV all drop to near 0 by 8–16× compression. Latent Compaction keeps ~57–82% at 16× across 4K–16K (Fig 1(d,e)). The same pattern holds for Qwen3-4B and 8B (Fig 4), and for the NIAH multi-key, multi-query, and multi-value RULER subtasks (Figs 12–16). This matters for the decision because it is the paper's main evidence that optimizing the cache, rather than selecting from it, can store information that selection loses. The gap is too large to be explained by noise alone.
Critical weaknesses
  1. The reported numbers do not agree with each other, and the main number cannot be found. The default setting (Qwen3-1.7B, Phonebook, 16×) is reported as:
  • 87.75 / 83.75 / 70.50 at 4K / 8K / 16K in Tables 1, 2, and 3 (average 80.67);
  • about 82 / 61 / 57 in Fig 1(e);
  • about 83 / 60 / 55 at the default 75% preserver point in Fig 5 (left); 61.00% for 1,000 + 1,000 steps in Fig 6 (model, ratio, and context length not stated; it does not match the default's table values of 87.75 / 83.75 / 70.50, average 80.67);
  • 43.75 / 32.25 / 24.25 for "Latent Compaction" in Table 4 (model not stated). This row is identical to Table 2's "Preserver KVs are trainable in stage 1" row.
  • The 8K gap between Table 1 and Fig 1(e), 83.75 vs. ~61, is far larger than any plot-reading error. As a result, I cannot tell which numbers describe the proposed method, or whether the ablations were run on the same configuration as the main results?

Separately, the abstract claims "96.7% accuracy at 32× compression, compared with 40.0% for KVZip+" on RULER. Neither number appears in the body or the appendix. Aggregate RULER in Fig 1(c) peaks at about 64% for the method and about 25% for KVZip+ at 32×. If 96.7% refers to a subset of subtasks or to one context length, the abstract should say so. This matters for the decision because the ablations are the paper's evidence for its design choices, and the abstract is its headline claim. Neither can be trusted as written.

  1. The comparison is matched on memory but not on compute, and the closest prior method is missing. The method spends 480 teacher rollouts, 1,000 updates per layer (28 layers) in stage 1, and 1,000 end-to-end updates of 4 rollouts each in stage 2, per document (Table 6). Most baselines are training-free selection methods. Yet no wall-clock time, GPU memory, or FLOPs for preparation are reported; the conclusion only says that fitting "remains costly." Cartridges (Eyuboglu et al., 2025), which the paper itself describes as training a compact KV cache through self-study and context distillation (§2), is the closest prior method and is not included as a baseline. Stage 2 of Latent Compaction closely follows that recipe. The Attention Matching baseline uses a specific "highest-attention-key variant, without on-policy query recollection or OMP" (App. A.3). It is close to 0 on Phonebook even at 4× (Fig 1(d)), and it is missing from LongBench (Fig 3) and from the cross-model results (Fig 4). Outside synthetic retrieval the picture is mixed. On LongBench, gains are smaller and depend on the model: with Qwen3-1.7B, KVZip+ is ahead at 4× (~31.5 vs. ~29), about equal at 8×, and the method leads only at 16–32× (Fig 3). This matters for the decision because the paper's contribution is a trade: much more preparation compute for much better accuracy. Without the cost numbers and the Cartridges baseline, readers cannot judge whether the trade is worthwhile.
Questions for authors

1- Which numbers describe the default method at Qwen3-1.7B / Phonebook / 16×: Tables 1–3 (87.75 / 83.75 / 70.50), Fig 1(e) (~82 / ~61 / ~57), Fig 5, or Fig 6 (61.00%)? Why does Table 4's "Latent Compaction" row equal Table 2's "Preserver KVs are trainable in stage 1" row? Were the ablations run with a different configuration, set of documents, or code version than the main results?

2- Where does the abstract's "96.7% vs. 40.0% at 32×" on RULER come from (which subtasks, context length, and model)?

3- What is the preparation cost per document (wall-clock time, GPU type, peak memory) for each context length, compared with KVZip+ and Attention Matching? How many reused queries are needed before this cost pays off?

4- How does the method compare with Cartridges at the same KV budget, and ideally at a similar training budget? If a comparison is not possible, why not?

5- Is the Attention Matching variant used here the one its authors recommend for this setting? Did you try Attention Matching with the same 480-prompt teacher queries your stage 1 uses? That would separate the effect of the objective from the effect of gradient-based fitting and your initialization.

6- Were the default fractions and step counts chosen on the same Phonebook documents used for the reported results? If so, can you report results on fresh documents? 7- How many Phonebook documents are used per cell? and what is the variance across documents and seeds for the main comparisons?

8- How is the text-summary baseline produced (model, prompt, length control)? On NIAH-Single-2 at 16K it scores ~10% at 16× but ~95% at 128× (Fig 10). What explains this?

9- Which LongBench subsets, how many examples, and what truncation rule were used? Can you add Attention Matching, SnapKV, and PyramidKV to Fig 3 and Fig 4?

2 comments
AuthorsOpenReview ↗

Rebuttal part 1/2

Dear Reviewer 4idX,

Thank you for the time and care you put into reviewing our paper and checking the results across figures and tables.

1. Inconsistent numbers and the headline RULER result

Abstract. The 96.7% versus 40.0% result was for the three NIAH single-needle subtasks. Our wording implied that it covered the full RULER suite. The revised abstract reports 71.0% for Latent Compaction versus 41.5% for KVZip+ on RULER with Qwen3-1.7B at 16K context and 16× compression.

Figure 1 and Tables 1–3. Figure 1(e) showed earlier results. In the revised figure, the Qwen3-1.7B Phonebook scores at 16× are 87.75 / 83.75 / 70.50% at 4K, 8K, and 16K, respectively, as in Tables 1–3. These runs use 1,000 updates in each stage and freeze preserver keys, values, and attention biases during Stage 1.

Figure 5. We replaced the submitted allocation sweep. At 16× compression, 75% preservers and 25% anchors within the remaining slots each give the highest recall at every tested length in their respective sweeps. Both settings reach 87.75 / 83.75 / 70.50% at 4K, 8K, and 16K, averaging 80.67%, consistent with the reference scores in Tables 1–3. See the updated Figure 5 values.

Figure 6. The score changes from 61.00 to 61.25 when Stage 1 increases from 1,000 to 2,000 updates with Stage 2 fixed at 1,000. The full training curves use an 8K context at 16× compression. We corrected the y-axis label and specified the setting in the caption.

Table 4. We mistakenly let preserver KVs train during Stage 1, which explains why the original reference row matched that ablation in Table 2. The revised table uses 1,000 updates in each stage and freezes preserver keys, values, and biases during Stage 1.

Method4K (%)8K (%)16K (%)32K (%)
Latent Compaction87.7683.7570.5060.50
↳ Remove attention biases (bcmpb_\mathrm{cmp})80.5063.0061.3348.50
↳ Equal mix, replacing Eq. 5 with 12(Acmp+Anew)\frac{1}{2}(A_\mathrm{cmp} + A_\mathrm{new})8.501.00.250.25

Removing attention biases lowers recall at every length. Equal mixing brings recall down to 8.50% or less.

Evaluation split and variability. Each original Phonebook comparison used 20 documents with 20 withheld lookups each. We chose the hard allocation for the anchor sweep on separate development documents. At 16K and 16×, Qwen3-4B and 8B improve over KVZip+ by 64.25 and 69.25 points, respectively. The paired 95% document-bootstrap intervals are [58.75, 69.75] and [63.75, 74.75]. These intervals capture variation across documents.

2. Preparation cost, closest baselines, and benchmark coverage

Cost and reuse. On one 80 GB NVIDIA A800, preparing a 16K Qwen3-4B document at 16× took 27.0 min (11.2 min for teacher QA generation and replay, 15.8 min for fitting). Serving took 2.9 vs. 9.5 s per query at batch 1 and 15.6 vs. 52.0 s per batch at batch 16, against a reused full KV cache with FlashAttention. Including all preparation, compaction breaks even after 246 serial queries, 720 queries at batch 16, or about 550 queries at each method's fastest measured batch size. The A800 measurements cover one context length. Comparing preparation time and peak memory with KVZip+ and Attention Matching across lengths remains open.

Break-even against KVZip+. On 16K RULER with Qwen3-1.7B, Latent Compaction at 32× scores 62.49, versus 57.74 for KVZip+ at 8×, using one quarter of its logical document KV slots. Preparation takes 23.4 min vs. 2.3 s and serving at batch size 1 takes 1.9 s vs. 15.2 s per query. Including preparation for both methods, Latent Compaction first becomes faster at 106 queries per document. See the cumulative timing calculation.

Latent Compaction is a proof of concept for fitting reusable document caches at test time. Batching teacher generation and replay could reduce preparation time, as could shorter fitting schedules. We leave these optimizations to future work.

AuthorsOpenReview ↗

Rebuttal part 2/2

Cartridges. We matched the KV budget at each compression ratio and reran Cartridges with 68× as much training data and 68× as many iterations as in our initial run. This roughly matches the official implementation's training scale. At 16×, this raises Cartridges from 17.59 to 56.47 on RULER and 1.00% to 19.50% on Phonebook. Latent Compaction reaches 70.97 and 70.50%, respectively, and leads both Cartridges runs at every tested ratio from 2× to 64×. See the 16K baseline comparison. Cartridges

Attention Matching. Our original baseline uses the upstream highest-attention-key solver with repeat-prefill queries and uniform allocation. The authors also evaluate repeat-prefill on RULER. With the same 480 teacher prompts as Latent Compaction, Attention Matching scores 18.75 on RULER and 0.00% on Phonebook at 16×. The gap persists with the prompts held fixed. Comparisons with the on-policy and OMP variants and at matched preparation time remain open. Attention Matching

LongBench. The Qwen3-1.7B comparison covers 14 English and two code tasks, with 20 fixed examples per task. We cap inputs at 16,384 tokens by trimming the middle of the source and keeping both ends and the full query. With Attention Matching, SnapKV, and PyramidKV included, Latent Compaction leads at 16× with 22.51, ahead of 21.10 for adapted SnapKV and 18.65 for KVZip+. KVZip+ leads at 4× and 8×, and adapted SnapKV leads at 32×. The revised appendix includes the full comparison.

Figure 4. Running every baseline across all models, context lengths, and compression ratios would be too costly. We show the full-cache reference and the strongest compressed baselines from our broader Qwen3-4B and 8B Phonebook sweep. In the settings covered by both evaluations, KVZip+ leads at 8× and 16×, and text summarization leads at 32×. The revised appendix includes the completed 4B and 8B comparisons with Attention Matching, SnapKV, and PyramidKV.

Text summary. Qwen3-1.7B greedily generates a factual summary from the source alone. The prompt states the token budget, and we truncate the output to that limit. We reproduced the NIAH-Single-2 reversal at 16K on the same documents (2/20 at 16× and 19/20 at 128×). Each budget generates a new summary. We have yet to determine why the shorter summaries worked better on these examples.

Thank you again for checking these details so carefully. We’d be happy to discuss any remaining questions. If the clarifications and new results address your concerns, we hope you’ll consider raising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.