- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Neural Context Compiler (NCC), a method for compressing long documents into reusable KV caches through document-specific optimization while keeping the underlying language model frozen. Starting from Attention Matching (AM) initialization, NCC refines compact keys and values by distilling full-context teacher predictions on document continuations, avoiding the synthetic question-answer generation required by self-study approaches. Experiments on Qasper, LongBench, and multiple language models demonstrate improvements in information recall and QA performance over existing KV-cache compression baselines, although the gains vary considerably across model architectures.
Critical strengths
- Effective document-specific KV cache refinement.
NCC introduces a conceptually simple approach that leverages the document itself as a supervision signal to refine compact KV caches without modifying the underlying language model. The improvements over AM in number recall and downstream QA, particularly on long documents, demonstrate the potential of output-level supervision for preserving information in compressed representations.
- Thorough experimental evaluation and analysis.
The paper provides extensive evaluations across compression budgets, document lengths, and model architectures, accompanied by carefully designed ablations and statistical analyses. The investigation of supervision coverage and abstention behavior is particularly informative, offering useful insights into why source-continuation training can outperform synthetic QA-based supervision in recovering document details.
Critical weaknesses
- Limited practical applicability due to substantial compilation overhead.
Despite the reported efficiency improvements over self-study, NCC still requires expensive document-specific optimization, with the standard pipeline taking approximately 20–41 minutes per document and requiring hundreds to thousands of queries to amortize its cost against repeated full-context prefilling. Although a generation-free variant reduces this overhead, the absence of end-to-end comparisons against practical alternatives such as reusable or quantized full-context KV caches and retrieval-based systems makes it difficult to establish whether NCC offers a compelling efficiency–accuracy tradeoff in realistic deployment scenarios.
- Inconsistent improvements across model architectures.
While NCC achieves substantial gains on the primary Qwen3.6-27B model, its improvements are considerably smaller on other architectures, and the results on Qwen3-8B and Olmo Hybrid 7B show degraded QA performance relative to AM. These results raise concerns about the general applicability of the approach, particularly because the main experiments rely on a hybrid architecture whose full-context teacher exhibits exceptionally strong numerical recall.
Questions for authors
- Practical efficiency and deployment scenarios
The reported compilation cost is substantial, particularly relative to the modest per-query latency difference between cached full-context inference and NCC. Could the authors clarify the target deployment scenarios in which this compilation overhead is justified, and provide comparisons against practical alternatives such as quantized KV caching, KV-cache offloading, or retrieval-augmented generation under comparable memory, latency, and accuracy constraints?
- Generalization across architectures
The experiments show that NCC does not consistently improve constrained recall or QA performance on pure-attention and certain hybrid models, despite reducing abstention rates. Can the authors explain which model characteristics or teacher capabilities determine whether source-continuation refinement improves actual information retrieval rather than simply increasing the model's willingness to generate answers?
4 comments
Response to Reviewer EMpK (2/2): model architectures (W2, Q2)
Your second question, whether refinement improves retrieval or only the willingness to answer, is the right way to read our results across models, and the answer below is organized around it. Table and figure numbers are the submission's.
W2. Improvements differ across architectures. You are right, and the revision states the scope plainly: NCC is a method for hybrid models. With a learning rate set per model, the gain in constrained number recall over AM replicates on four of five hybrids, from two families: +17.5 [14.7, 20.5] points on the main model and +7.8 to +10.2 on Qwen3.5-9B and two Nemotron models (every interval above zero; Table 7). QA F1 rises on the main model (+6.1 [2.4, 9.7]) and is not distinguishable from AM's on those three (+2.8 [−1.6, 7.3], −0.8 [−4.6, 2.5], −0.4 [−4.4, 3.6]). On Olmo Hybrid 7B and on the one pure-attention model we tested, Qwen3-8B, refinement does not improve on AM (recall −1.4 [−5.0, 2.1] and +1.8 [−3.1, 6.7]; QA F1 −6.8 [−11.5, −2.3] and −5.5 [−10.9, −0.1]), and AM alone is the better choice there. In the revised PDF the six-model table and the pure-attention result are in Section 5.2 of the main text, and Appendices B and C report every learning-rate setting that was run.
Q2. What decides whether refinement improves retrieval rather than the willingness to answer?
- On the main model the gain is retrieval. Number-constrained decoding makes every state answer with a number, so the willingness to answer plays no role. Under it, correct answers rise from 557 to 720 of 916, and the first number token agrees with the full-context reader's on 90.5% of the questions instead of 79.4% (Table 17).
- The retrieved content is in the refined KV entries. In a new experiment on the main model (Appendix Z.1 of the revised PDF) we replaced the document's recurrent state at read time with that of an empty document. The gain over AM is intact: +20.3 [17.1, 23.6] points of constrained recall, against +17.8 [14.6, 21.0] with the full-document state in the same run.
- The full-context reader is what is distilled. NCC trains the cache to predict the document's text as the full-context reader does, so what it can pass on depends on how well that reader predicts the text. Among the five hybrids the gain follows the full-context reader's recall (99.9% and 99.0%: +17.5 and +10.2 points; 67%: +9.7 and +7.8; 53.6%: −1.4; Table 7). On the digits of the document, the main model's reader has a loss of 0.004 nats and Qwen3-8B's 0.30 nats (Appendix C.1). This bears on your remark about the main model's teacher: the method works best with an accurate full-context reader.
- A label-free check for a new model. The number probe is built from the document's own text and needs no annotation. Before adopting the refinement on a new model, one can compile a few documents and compare the refined cache with AM under constrained decoding, on recall and on the agreement of the first token with the full-context reader. In our six models the agreement goes from 79.4% to 90.5% on the main model and from 77.3% to 86.7%, 61.7% to 70.1% and 64.8% to 68.7% on the three further models where recall improves, and does not rise on the remaining two (Figure 10b; Table 11; Appendix B.2).
We would be glad to answer further questions.
Gentle reminder for Reviewer EMpK
Dear Reviewer EMpK,
Thank you again for your review and for the time you have given to our submission. We understand that this is a busy time. If time permits before the discussion period closes, we would be grateful to know whether our two replies above address your concerns: the compile cost and the deployment alternatives (W1, Q1), and the results across model architectures and what the refinement improves (W2, Q2). If any point remains open, we would be glad to clarify it.
Thank you for your time and consideration.
The authors
Thank you for the detailed rebuttal and additional experiments.
I agree with Reviewer P9m7 that the extensive revisions and new experiments are difficult to fully evaluate within the short discussion period. The revised paper is also quite dense, so I may have missed or misunderstood some of the results.
While the memory savings are clear, I still have concerns about the accuracy tradeoff, especially for long documents. For example, on 16-paper documents, NCC achieves 34.0 QA F1 using 204 MiB, compared with 44.8 F1 using 1,538 MiB for a quantized full-context cache. I wonder whether such a substantial accuracy gap is justified by the memory savings in practical settings.
Regarding generalization, I understand that NCC targets hybrid models. However, if I understand the results correctly, only Qwen3.6-27B shows a statistically significant improvement in QA F1 (+6.1), while Olmo Hybrid 7B shows a significant degradation (−6.8) in table 2 of the revise draft. This makes me question how consistently NCC benefits downstream QA, even across hybrid models.
Please correct me if I have misinterpreted any of these results. I appreciate the additional evidence, but I still have reservations about these two points and will maintain my score of 2 for now.
Response to Reviewer EMpK (1/2): compile cost and deployment (W1, Q1)
Thank you for the careful review. Your question about compilation cost led us to measure the generation-free pipeline on long documents. Table numbers below are the submission's; Appendix Z is in the revised PDF (uploaded, changes in blue); "KV entries" are the submission's "rows".
W1. Compile cost. Most of the 20–41 minutes you cite is spent generating the text on which AM is fitted, and this step can be left out. Without any generated text, a 34K–51K-token document compiles in a median of 587 s and a 76K–97K-token document in 957 s (512 KV entries per KV head; about 10 and 16 minutes; 40% and 39% of the standard pipeline's time; a single paper: 242 s, Table 24). Against prefilling the document for every query, compilation then pays for itself after 121 and 89 queries (standard pipeline: 291 and 224). In accuracy, on 16-paper documents the generation-free pipeline is not distinguishable from the standard one (constrained number recall −0.3 [−2.2, 1.5]; QA F1 +2.7 [−1.5, 7.1]), and on 8-paper documents it is not distinguishable from a rerun of the standard pipeline (−0.7 [−3.2, 1.8]; −0.8 [−5.4, 4.0]). Details are in Appendix Z.3.
Q1. Where the overhead is justified. The target is a server that keeps many long documents resident on a memory-bound GPU, each queried on the order of a hundred times or more. Compilation is a one-time offline job. After it, a 76K–97K-token document reaches its first token after 0.12 s instead of 11.1 s with per-query prefill; 361 compiled 512-entry states fit on one GPU beside the weights, where 7 full caches fit; and in our serving stack compiled states decode in batches of 128 requests, where full caches run out of memory at 8 (Tables 22 and 23).
Q1. Comparison with practical alternatives. Size of the stored state, and constrained number recall (%) / QA F1 (single papers: Table 21; 16-paper documents: Tables 1 and 22):
"–": not run. The 4-bit values for 16-paper documents are from Appendix E.1 of the revised PDF. The 4-bit states were not timed.
Appendix F of the revised PDF gives the full comparison at all three lengths, with latency and states per GPU.