acceptodds
Under review as a conference paper at ICLR 2027

Neural Context Compiler: Learning KV Caches for Long-Context LLMs

Abstract

Documents that are queried many times can be compiled once into a compact key–value (KV) cache that a frozen language model reads at every query. We show that such compaction must alter attention at almost every query and that a training objective by itself bounds the loss only on the inputs it supervises. The leading learned method, self-study, supervises answers to synthetic questions that must first be generated for each document, and hence only the facts those answers state. We introduce Neural Context Compiler (NCC), whose refinement supervises a prediction of every document token and generates no text: starting from an Attention Matching (AM) cache, it refines only the compact keys and values by distilling the full-context teacher on teacher-forced continuations of the document. With the hybrid Qwen3.6-27B on 64 held-out papers, a 512-row NCC cache raises constrained recall of masked source numbers from 62.3% to 79.8% over AM and Qasper F1 from 38.1 to 44.2, 28–44 points above StreamingLLM, SnapKV and KVzip. It is statistically indistinguishable from self-study on constrained recall and QA, 11 points better under greedy decoding and 13–25 points better on numbers self-study’s answers never state, with 92% less refinement time. On three of four LongBench QA tasks it matches the full-context reader with a 5–11×smaller state, and on 34K–97K-token documents at 11–30×compression it retains 74–84% of full-context QA F1 (AM: 19–49%). A fully generation-free variant does as well, an FP4 reader with a 4-bit state keeps its accuracy with a 14×smaller stored state, and the recall gain replicates on three further hybrids, fading where the teacher misses many numbers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long-context language models, learned KV caches, context distillation, factual recall, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_EMpKOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Neural Context Compiler (NCC), a method for compressing long documents into reusable KV caches through document-specific optimization while keeping the underlying language model frozen. Starting from Attention Matching (AM) initialization, NCC refines compact keys and values by distilling full-context teacher predictions on document continuations, avoiding the synthetic question-answer generation required by self-study approaches. Experiments on Qasper, LongBench, and multiple language models demonstrate improvements in information recall and QA performance over existing KV-cache compression baselines, although the gains vary considerably across model architectures.

Critical strengths
  1. Effective document-specific KV cache refinement.

NCC introduces a conceptually simple approach that leverages the document itself as a supervision signal to refine compact KV caches without modifying the underlying language model. The improvements over AM in number recall and downstream QA, particularly on long documents, demonstrate the potential of output-level supervision for preserving information in compressed representations.

  1. Thorough experimental evaluation and analysis.

The paper provides extensive evaluations across compression budgets, document lengths, and model architectures, accompanied by carefully designed ablations and statistical analyses. The investigation of supervision coverage and abstention behavior is particularly informative, offering useful insights into why source-continuation training can outperform synthetic QA-based supervision in recovering document details.

Critical weaknesses
  1. Limited practical applicability due to substantial compilation overhead.

Despite the reported efficiency improvements over self-study, NCC still requires expensive document-specific optimization, with the standard pipeline taking approximately 20–41 minutes per document and requiring hundreds to thousands of queries to amortize its cost against repeated full-context prefilling. Although a generation-free variant reduces this overhead, the absence of end-to-end comparisons against practical alternatives such as reusable or quantized full-context KV caches and retrieval-based systems makes it difficult to establish whether NCC offers a compelling efficiency–accuracy tradeoff in realistic deployment scenarios.

  1. Inconsistent improvements across model architectures.

While NCC achieves substantial gains on the primary Qwen3.6-27B model, its improvements are considerably smaller on other architectures, and the results on Qwen3-8B and Olmo Hybrid 7B show degraded QA performance relative to AM. These results raise concerns about the general applicability of the approach, particularly because the main experiments rely on a hybrid architecture whose full-context teacher exhibits exceptionally strong numerical recall.

Questions for authors
  1. Practical efficiency and deployment scenarios

The reported compilation cost is substantial, particularly relative to the modest per-query latency difference between cached full-context inference and NCC. Could the authors clarify the target deployment scenarios in which this compilation overhead is justified, and provide comparisons against practical alternatives such as quantized KV caching, KV-cache offloading, or retrieval-augmented generation under comparable memory, latency, and accuracy constraints?

  1. Generalization across architectures

The experiments show that NCC does not consistently improve constrained recall or QA performance on pure-attention and certain hybrid models, despite reducing abstention rates. Can the authors explain which model characteristics or teacher capabilities determine whether source-continuation refinement improves actual information retrieval rather than simply increasing the model's willingness to generate answers?

4 comments
AuthorsOpenReview ↗

Response to Reviewer EMpK (1/2): compile cost and deployment (W1, Q1)

Thank you for the careful review. Your question about compilation cost led us to measure the generation-free pipeline on long documents. Table numbers below are the submission's; Appendix Z is in the revised PDF (uploaded, changes in blue); "KV entries" are the submission's "rows".

W1. Compile cost. Most of the 20–41 minutes you cite is spent generating the text on which AM is fitted, and this step can be left out. Without any generated text, a 34K–51K-token document compiles in a median of 587 s and a 76K–97K-token document in 957 s (512 KV entries per KV head; about 10 and 16 minutes; 40% and 39% of the standard pipeline's time; a single paper: 242 s, Table 24). Against prefilling the document for every query, compilation then pays for itself after 121 and 89 queries (standard pipeline: 291 and 224). In accuracy, on 16-paper documents the generation-free pipeline is not distinguishable from the standard one (constrained number recall −0.3 [−2.2, 1.5]; QA F1 +2.7 [−1.5, 7.1]), and on 8-paper documents it is not distinguishable from a rerun of the standard pipeline (−0.7 [−3.2, 1.8]; −0.8 [−5.4, 4.0]). Details are in Appendix Z.3.

Q1. Where the overhead is justified. The target is a server that keeps many long documents resident on a memory-bound GPU, each queried on the order of a hundred times or more. Compilation is a one-time offline job. After it, a 76K–97K-token document reaches its first token after 0.12 s instead of 11.1 s with per-query prefill; 361 compiled 512-entry states fit on one GPU beside the weights, where 7 full caches fit; and in our serving stack compiled states decode in batches of 128 requests, where full caches run out of memory at 8 (Tables 22 and 23).

Q1. Comparison with practical alternatives. Size of the stored state, and constrained number recall (%) / QA F1 (single papers: Table 21; 16-paper documents: Tables 1 and 22):

StateSingle paper: MiBRecall / F116-paper document: MiBRecall / F1
Full cache, BF16415.699.9 / 45.85,44099.9 / 44.1
Full cache, 4-bit116.999.9 / 46.21,537.699.9 / 44.8
NCC, 512 entries107.879.3 / 45.8107.833.0 / 21.8
NCC, 2,048 entries––204.044.0 / 34.0
NCC, 512 entries, 4-bit30.478.8 / 44.830.431.9 / 22.7

"–": not run. The 4-bit values for 16-paper documents are from Appendix E.1 of the revised PDF. The 4-bit states were not timed.

  • Quantized caches. Compaction and quantization compose. Quantization shrinks every entry by a fixed factor (3.6× for the whole state at 4 bits). Compaction removes entries, so its factor grows with the document: 26.7× to 50.5× at 76K–97K tokens, and 180× when the compact state is also stored in 4 bits (recall 31.9%, against 32.7% for the BF16 state in the same study; Appendix E.1).
  • A cached full context. Against a full cache that is already resident, compilation is not repaid in time: the first token arrives 0.10 s earlier (0.12 s instead of 0.22 s), and the speed per token is about the same. The benefit there is memory: 5,440 MiB against 108–204 MiB per document and, in our serving stack, 7.31 GiB against 0.16–0.36 GiB of GPU memory per request.
  • Offloading. Served from host memory, a full 16-paper state reaches its first token after 0.239 s and, in our stack, still needs 7.31 GiB of GPU memory per request; a compiled state needs 0.120 s and 0.16 GiB (Appendix F).
  • Retrieval. Retrieval is complementary: it selects text that is read again at every query, whereas a compiled state is read without re-reading text. We did not compare with it.

Appendix F of the revised PDF gives the full comparison at all three lengths, with latency and states per GPU.

AuthorsOpenReview ↗

Response to Reviewer EMpK (2/2): model architectures (W2, Q2)

Your second question, whether refinement improves retrieval or only the willingness to answer, is the right way to read our results across models, and the answer below is organized around it. Table and figure numbers are the submission's.

W2. Improvements differ across architectures. You are right, and the revision states the scope plainly: NCC is a method for hybrid models. With a learning rate set per model, the gain in constrained number recall over AM replicates on four of five hybrids, from two families: +17.5 [14.7, 20.5] points on the main model and +7.8 to +10.2 on Qwen3.5-9B and two Nemotron models (every interval above zero; Table 7). QA F1 rises on the main model (+6.1 [2.4, 9.7]) and is not distinguishable from AM's on those three (+2.8 [−1.6, 7.3], −0.8 [−4.6, 2.5], −0.4 [−4.4, 3.6]). On Olmo Hybrid 7B and on the one pure-attention model we tested, Qwen3-8B, refinement does not improve on AM (recall −1.4 [−5.0, 2.1] and +1.8 [−3.1, 6.7]; QA F1 −6.8 [−11.5, −2.3] and −5.5 [−10.9, −0.1]), and AM alone is the better choice there. In the revised PDF the six-model table and the pure-attention result are in Section 5.2 of the main text, and Appendices B and C report every learning-rate setting that was run.

Q2. What decides whether refinement improves retrieval rather than the willingness to answer?

  1. On the main model the gain is retrieval. Number-constrained decoding makes every state answer with a number, so the willingness to answer plays no role. Under it, correct answers rise from 557 to 720 of 916, and the first number token agrees with the full-context reader's on 90.5% of the questions instead of 79.4% (Table 17).
  2. The retrieved content is in the refined KV entries. In a new experiment on the main model (Appendix Z.1 of the revised PDF) we replaced the document's recurrent state at read time with that of an empty document. The gain over AM is intact: +20.3 [17.1, 23.6] points of constrained recall, against +17.8 [14.6, 21.0] with the full-document state in the same run.
  3. The full-context reader is what is distilled. NCC trains the cache to predict the document's text as the full-context reader does, so what it can pass on depends on how well that reader predicts the text. Among the five hybrids the gain follows the full-context reader's recall (99.9% and 99.0%: +17.5 and +10.2 points; 67%: +9.7 and +7.8; 53.6%: −1.4; Table 7). On the digits of the document, the main model's reader has a loss of 0.004 nats and Qwen3-8B's 0.30 nats (Appendix C.1). This bears on your remark about the main model's teacher: the method works best with an accurate full-context reader.
  4. A label-free check for a new model. The number probe is built from the document's own text and needs no annotation. Before adopting the refinement on a new model, one can compile a few documents and compare the refined cache with AM under constrained decoding, on recall and on the agreement of the first token with the full-context reader. In our six models the agreement goes from 79.4% to 90.5% on the main model and from 77.3% to 86.7%, 61.7% to 70.1% and 64.8% to 68.7% on the three further models where recall improves, and does not rise on the remaining two (Figure 10b; Table 11; Appendix B.2).

We would be glad to answer further questions.

AuthorsOpenReview ↗

Gentle reminder for Reviewer EMpK

Dear Reviewer EMpK,

Thank you again for your review and for the time you have given to our submission. We understand that this is a busy time. If time permits before the discussion period closes, we would be grateful to know whether our two replies above address your concerns: the compile cost and the deployment alternatives (W1, Q1), and the results across model architectures and what the refinement improves (W2, Q2). If any point remains open, we would be glad to clarify it.

Thank you for your time and consideration.

The authors

Thank you for the detailed rebuttal and additional experiments.

I agree with Reviewer P9m7 that the extensive revisions and new experiments are difficult to fully evaluate within the short discussion period. The revised paper is also quite dense, so I may have missed or misunderstood some of the results.

While the memory savings are clear, I still have concerns about the accuracy tradeoff, especially for long documents. For example, on 16-paper documents, NCC achieves 34.0 QA F1 using 204 MiB, compared with 44.8 F1 using 1,538 MiB for a quantized full-context cache. I wonder whether such a substantial accuracy gap is justified by the memory savings in practical settings.

Regarding generalization, I understand that NCC targets hybrid models. However, if I understand the results correctly, only Qwen3.6-27B shows a statistically significant improvement in QA F1 (+6.1), while Olmo Hybrid 7B shows a significant degradation (−6.8) in table 2 of the revise draft. This makes me question how consistently NCC benefits downstream QA, even across hybrid models.

Please correct me if I have misinterpreted any of these results. I appreciate the additional evidence, but I still have reservations about these two points and will maintain my score of 2 for now.

Reviewer_UJa6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Neural Context Compiler (NCC), a method that compiles a document into a compact KV cache for repeated querying. NCC starts from an Attention Matching cache and refines its keys and values by distilling a full-context teacher on teacher-forced continuations of the source document. This avoids generating synthetic question-answer pairs for refinement. NCC improves over AM in the main hybrid-model setting and achieves performance comparable to self-study on several metrics with substantially lower refinement cost.

Critical strengths

The method addresses a practical problem. Compiling a frequently queried document into a compact native cache can reduce the state needed for serving while keeping the reader unchanged.

The method is easy to understand and the paper is well-organized.

Critical weaknesses

My main concern is limited validation under distribution shift. Although some QA benchmarks have demonstrated the performance of NCC, its effect on other domain/scenario is still unclear.

Could you provide more detailed cases or an analysis of cases where NCC fails?

Questions for authors

Why is the reference in a two-column format?

3 comments
AuthorsOpenReview ↗

Response to Reviewer UJa6

Thank you for the review. Your concern about distribution shift led us to run a new benchmark with another kind of text and another task, and your request for failure cases led to a new appendix. A revised PDF is uploaded (changes in blue). Table numbers below are the submission's.

W1. Validation under distribution shift.

New: meeting transcripts and summarization. On LongBench QMSum (query-focused summarization of 35 meeting transcripts, 200 queries; recipe unchanged; Appendix Z.5 of the revised PDF), NCC scores 18.2 ROUGE-L against 12.4 for AM (+5.8 [4.4, 7.4]) and 17.5 for the full-context reader (+0.7 [−0.9, 2.4]), with a 7.3× smaller state. The training-free selection baselines score 3.3–3.9. NCC answers 169 of the 200 queries, AM 114 and the full-context reader 148.

Already in the submission. The benchmarks cover five kinds of text. The recipe is the same in every row, and nothing is trained across documents. NCC − AM on the main model, with 95% intervals (Tables 1, 5, 6, 38, 40):

Kind of textBenchmarkTaskNCC − AM
NLP research papersQasper (64 papers)number recall; QA F1+17.5 [14.7, 20.5]; +6.1 [2.4, 9.7]
Fiction and articlesQuALITY (64 articles)multiple choice+0.2 [−1.6, 2.0]
Novels and movie scriptsNarrativeQA (20 stories, up to 68K tokens)QA F1+4.7 [1.7, 7.9]
Documents from mixed fieldsMultiFieldQA-en (100 documents)QA F1+12.4 [8.1, 16.9]
Wikipedia passages, multi-hopHotpotQA; 2WikiMQA (100 documents each)QA F1+21.4 [13.3, 29.7]; +33.1 [24.6, 42.0]

On NarrativeQA, HotpotQA and 2WikiMQA, NCC is also not distinguishable from the full-context reader (+0.6 [−3.3, 4.7], −1.1 [−7.0, 4.7] and −0.7 [−9.1, 7.7] F1) with a 5.1–11.4× smaller state. On 34K–97K-token documents the gain over AM is +13.1 to +25.2 QA F1 (every interval above zero; Table 1). The revised PDF gathers these results in a new Appendix X, "Results by kind of text".

W2. Cases, and an analysis of where NCC fails. The new Appendix Y, "Failure analysis", gives the analysis, with five worked cases that show the answer of every state. Three places where the method helps less:

  • Long exact numbers. A compact state can keep most digits of a number and lose one. In one case the document says 219362; the full-context reader returns 219362 and NCC returns 21936. On 34K–97K-token documents NCC recalls 33.0–57.6% of the masked numbers, where AM recalls 25.8–41.1% and the full context over 99% (Table 1).
  • Multiple choice. On QuALITY, NCC is level with AM (+0.2 [−1.6, 2.0]; AM 84.9%, NCC 85.1%, full context 90.6%): a single-letter decision is already largely kept by AM.
  • Pure-attention models. On Qwen3-8B, the one pure-attention model we tested, refinement does not improve on AM, and AM alone is the better choice there. The revision states this scope up front: NCC is a method for hybrid models, and with a learning rate set per model the recall gain over AM replicates on four of five of them (+7.8 to +17.5 points, every interval above zero; Table 7).

Q1. Why are the references in two columns? That was a formatting choice of ours. Thank you for pointing it out; the revised PDF uses the template's single-column format.

We would be glad to answer further questions.

AuthorsOpenReview ↗

Gentle reminder for Reviewer UJa6

Dear Reviewer UJa6,

Thank you again for your review and for the time you have given to our submission. We understand that this is a busy time. If time permits before the discussion period closes, we would be grateful to know whether our reply above addresses your points: the validation on other kinds of text, with the new results on meeting summarization (W1); the failure analysis in the new Appendix Y (W2); and the reference list, now in a single column (Q1). If anything remains open, we would be glad to clarify it.

Thank you for your time and consideration.

The authors

Reviewer_UJa6OpenReview ↗

Response

Thank you for your reply. I will maintain my score.

Reviewer_P9m7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Neural Context Compiler (NCC), which compiles a document once into a fixed-size KV cache that a frozen LLM reads for all later queries. NCC uses Attention Matching to initialize a compacted KV cache and refines it by distilling the original model on teacher-forced continuations of the document itself. Experiments mainly evaluate Qwen3.6-27B on 64 Qasper papers, QuALITY and four LongBench tasks. NCC outperforms AM and training-free selection, and matches a Cartridges-style self-study baseline on QA at lower cost.

Critical strengths
  1. Simplicity of method: Distilling the full-context teacher on teacher-forced document continuations is simple, needs no synthetic text generation, and consistently improves over Attention Matching on Qwen3.6-27B across budgets and document lengths up to 97K tokens.

  2. Informative ablations and reporting: The paper includes useful ablations (highlighting the role of initialization and teacher windowing) as well as realistic serving measurements. Furthermore, negative results on pure-attention models are reported.

Critical weaknesses
  1. Comparison to self-study is skewed toward NCC: The number recall task asks the model to fill in masked numbers from the document text, which is essentially what NCC is trained to do during continuation distillation, so it is unsurprising that NCC outperforms self-study here. On more realistic QA tasks like Qasper and NarrativeQA, NCC is mostly statistically indistinguishable from self-study. At the same time, the self-study control is scaled down compared to Cartridges [1] as its questions are generated from a few fixed document windows that cover only a fraction of long documents, and longer training reuses the same question bank instead of growing it. Finally, the claimed runtime savings rely on unbatched decoding for self-study and shrink considerably once the shared AM initialization is included.

  2. Gains appear tied to the hybrid architecture: Main experiments evaluate Qwen3.6-27B, where the full-context Gated DeltaNet state accounts for roughly 70% of the compact state. On pure-attention Qwen3-8B (Appendix C), NCC's advantage over AM essentially disappears. Mentioning this negative result only in a table caption and the appendix is problematic, as the title implies general applicability for LLM KV compression.

  3. Limited utility of Section 3: Section 3 mostly restates standard results (change of measure, KL chain rule, a KL margin for keeping the top token), and the authors note that applying Proposition 2 to the evaluation is a heuristic step. These results neither guide the method design nor predict anything that would not be expected without them. The space would be better used to explain AM (what is matched and how β\beta enters) and to move the pure-attention result into the main text. More generally, the manuscript is very dense, and the abstract and results read mostly as lists of numbers.

Nitpick: "Rows" is nonstandard for compact KV entries and is easily confused with query rows of the attention matrix. It should also be stated more prominently that only 451 of the 512 rows hold document tokens.

[1] https://arxiv.org/abs/2506.06266

Questions for authors
  1. Self-study at scale: How does self-study compare when its question bank grows (e.g., 4–16× more conversations, with windows sampled across the whole document, especially for long documents) and when generation uses a batched inference engine? A comparison at matched GPU time with batched generation would make the cost claim more robust.
  2. Role of the recurrent state: Is the GDN state ever reset or truncated in your experiments? What happens to NCC's gain if the refined KV rows are combined with the recurrent state of an empty or truncated document?
  3. Choice of budget: The compact size is fixed (512 rows per KV head) regardless of document length, while the number of spans is fixed at 128, so span length and training cost grow with the document. How should the budget be chosen for a given document, and should the number of spans scale with length?
5 comments
AuthorsOpenReview ↗

Response to Reviewer P9m7 (1/3): the recurrent state and the scope (W2, Q2)

Thank you for an expert and constructive review. Each of your questions led to an experiment that we ran during the discussion, and the paper has changed accordingly. We answer in three parts: the recurrent state and the scope (this part), self-study (part 2), the budget and the presentation (part 3). Table numbers are those of the submission; Appendix Z is in the revised PDF (uploaded, changes in blue); "KV entries" are the submission's "rows".

Q2 and W2(a). Is the recurrent state ever reset or truncated? What if it comes from an empty or truncated document?

Your question led to the most informative new experiment in the revision (Appendix Z.1). The gain over AM is intact without the document's recurrent state: the refined KV entries themselves carry the document.

In the submission the recurrent state is never reset or truncated. Every compact state, baselines included, keeps the state after the whole document, and it is, as you note, about 70% of the compact state (75.75 of 107.8 MiB). In the new experiment we re-read the stored states of the 64 papers (512 entries per KV head) and exchanged only the recurrent state at read time:

Recurrent state at read timeRecall, AMRecall, NCCNCC − AM, recallNCC − AM, QA F1
Full document (the paper's setting)61.679.4+17.8 [14.6, 21.0]+7.5 [3.3, 11.7]
First 50% of the document61.179.5+18.4 [15.5, 21.3]+6.8 [2.5, 11.0]
First 25% of the document60.879.2+18.4 [15.3, 21.5]+6.2 [2.5, 10.0]
A different paper59.478.9+19.5 [16.0, 23.1]+7.2 [3.2, 11.2]
Empty document (chat template only)56.476.8+20.3 [17.1, 23.6]+16.9 [12.5, 21.6]

Recall is constrained number recall (%); intervals are 95%. The first row re-reads the stored states of Table 21 (62.0 and 79.3 there).

We also refined the entries beside the empty-document state and read them with it, so that nothing of the document is in the compact state except its KV entries. This state reaches 78.2% recall and 42.1 QA F1, against 79.4% and 45.9 in the paper's setting (differences −1.2 [−3.6, 1.1] and −3.7 [−8.0, 0.5]). For comparison, simply reading with the empty-document state lowers QA F1 for AM, for NCC and even for the exact full KV cache (full cache 45.9 → 29.5; AM 38.4 → 20.0; NCC 45.9 → 36.9; every interval below zero); entries refined beside that state recover 5.2 F1 [1.2, 9.1] (36.9 → 42.1). The experiment was run on the main model.

W2(b). The pure-attention model. You are right that the scope has to be stated up front, and the revision does so: NCC is a method for hybrid models. With a learning rate set per model, the recall gain over AM replicates on four of five hybrids, from two families (+7.8 to +17.5 points, every interval above zero; Table 7). On the one pure-attention model we tested, Qwen3-8B, refinement does not improve on AM (NCC − AM: +1.8 [−3.1, 6.7] points of constrained recall, −5.5 [−10.9, −0.1] QA F1), and AM alone is the better choice there.

W2(c). Where this is reported; the title. In the revised PDF the abstract and the introduction are written around hybrid long-context models and state this scope; the six-model table and the pure-attention result are in Section 5.2 of the main text; a Limitations paragraph states the scope again; and Appendices B and C report every learning-rate setting that was run. The title is unchanged in this revision. If you find it helpful, we would be glad to add "Hybrid" to the title for the final version.

AuthorsOpenReview ↗

Response to Reviewer P9m7 (2/3): the comparison with self-study (W1, Q1)

Your points on the self-study control were fair, and they led to two additions to the paper: a control that covers the whole document with a larger bank (Appendix J), and a comparison at equal updates and at equal GPU time (Appendix Z.4). Against this stronger control NCC holds up, with a refinement that generates no text.

W1(c) and Q1. A larger bank, with windows across the whole document. The original eight windows covered 39% and 19% of the tokens of 8- and 16-paper documents. The new control tiles the whole long document with 2,048-token windows, generates 32 conversations per window and uses a new conversation at every update, so that the bank grows with the document and with training: 544–800 conversations on 8-paper documents and 1,216–1,536 on 16-paper documents, 2–6× the original 256.

Q1. Equal updates and equal GPU time. We ran NCC for the same number of updates as the control on each document (the same 128 spans, more passes). We also evaluated an earlier checkpoint that takes, on average, no more GPU time than the control's update steps alone, that is, with its question generation and teacher passes counted as free. NCC minus the full-coverage control (recall: constrained number recall; the control's numbers are those of its original run):

Documents, KV entriesRecall, equal updatesQA F1, equal updatesRecall, equal GPU time
8 papers, 512−0.4 [−4.0, 2.8]+5.4 [0.8, 10.0]−2.2 [−5.3, 1.1]
16 papers, 512−2.2 [−5.7, 1.7]−0.7 [−6.3, 5.3]−1.7 [−5.5, 2.5]
8 papers, 2,048+8.9 [5.1, 12.6]+4.5 [0.1, 8.8]+5.8 [2.1, 9.6]
16 papers, 2,048+7.3 [3.0, 11.9]+7.8 [3.0, 12.9]+6.7 [2.0, 11.8]†

With equal updates NCC is ahead at 2,048 entries, in recall and in QA F1, and is not distinguishable from the control in recall at 512 entries. At equal GPU time it is still ahead in recall on 8-paper documents at 2,048 entries and not distinguishable at 512 entries. Intervals are over documents for 8-paper cells and over papers for 16-paper cells, as in the submission (†over the four documents: [−1.7, 12.2]). NCC's 256-update checkpoints in this run are within 1.6 points of recall of the submitted values.

At NCC's default 256 updates, where the control has 2–6× as many, the control recalls more numbers at 512 entries (by 3.4 [0.2, 6.6] and 4.7 [0.7, 8.3] points) and the two are not distinguishable at 2,048 entries (NCC − control: +3.6 [−1.4, 9.0] and +2.3 [−1.9, 6.9]).

Q1 and W1(d). Batched generation. The last column of the table holds for any decoding engine: it charges self-study nothing for generating questions and answers or for its teacher passes, and no generation engine can go below that. We did not time batched generation itself.

W1(d). Cost with the shared AM initialization. With the shared AM initialization counted, as you ask, the whole pipeline takes 1,247 s per paper for NCC against 2,756 s for self-study, that is 45% (refinement stage alone: 127 s against 1,635 s; Table 44). The revision gives both costs in the main text (Section 5.1). Independent of any engine, NCC's refinement decodes no text (0 against 19,176 generated tokens per paper) and needs no question bank. The initialization can also be made generation-free; the whole pipeline then compiles a 34K–51K-token document, at 512 entries, in a median of 587 s (Appendix Z.3).

W1(a), (b). Number recall is close to the training signal; on QA the two are on par. Number recall is indeed close to what NCC is trained on; the revision says so in Section 5.1 and states the QA gain first, there as well as in the abstract and the introduction. On single-document QA your summary is also our claim: NCC is not distinguishable from self-study (NCC − self-study: +0.8 F1 [−2.4, 4.2] on Qasper, +0.5 [−3.7, 5.3] on NarrativeQA), and its refinement generates no text. Free-form answers are not close to the training signal, and there NCC gains over AM: +6.1 F1 [2.4, 9.7] on Qasper, +4.7 to +33.1 on four LongBench tasks, +13.1 to +25.2 on 34K–97K-token documents (every interval above zero) and, new, +5.8 ROUGE-L [4.4, 7.4] on query-focused summarization (Appendix Z.5).

AuthorsOpenReview ↗

Response to Reviewer P9m7 (3/3): the budget, the spans and the presentation (Q3, W3, nitpick)

Q3. How should the budget be chosen, and should the number of spans scale with length? Your question led to a simple recipe for long documents (Appendix Z.2): let the budget grow in proportion to the document, keep the 128 spans, and add passes.

Budget. Across lengths, recall follows the K/V compression (cached tokens per KV entry; Tables 1 and 42) rather than the length itself:

K/V compressionSetting (entries per KV head)Constrained recall
≈5×single papers, 1,02493.7%
≈10×single papers, 51279.8%
≈21×single papers, 256; 8-paper documents, 2,04856.6%; 57.6%
≈42×single papers, 128; 16-paper documents, 2,04841.1%; 44.0%
≈43× (new)8-paper documents, 1,02444.2%
≈85×; ≈168×8-paper; 16-paper documents, 51237.9%; 33.0%

The new row is a test of this rule: from the two existing settings at about 42× we predicted 41–44% before the run, and we measured 44.2%. To choose a budget, one picks the ratio for the accuracy that is needed and divides the document length by it. At about 10×, about 80% of the exact numbers are kept and QA F1 is on par with the full context (−1.7 [−4.6, 1.2]; Table 40).

Spans. You are right that the span length grows with the document. We tested whether the spans should grow in number instead (8-paper documents, 2,048 entries). More passes over the same 128 spans help: with 1,024 updates instead of 256, recall rises from 56.2% (the default retrained in the same run) to 65.4% (+9.2 [6.5, 11.5]) and QA F1 from 38.7 to 43.0 (+4.3 [0.7, 7.5]; full context: 46.2). More, shorter spans do not help: with the spans scaled with the length (1,024 spans, the same two passes), recall is not distinguishable from the default (−3.5 [−9.8, 2.5]) and QA F1 is 7.5 lower [3.6, 11.2]; with 2,048 updates for both, 1,024 spans are 12.3 points of recall [8.5, 16.1] below 128 spans. The run of part 2 agrees: with more passes over the same spans, NCC's recall rises in all four settings, by 4.5 [2.9, 6.3] and 4.2 [1.1, 7.0] points on 8-paper documents and by 3.7 [0.9, 6.5] and 3.9 [0.7, 7.0] on 16-paper documents (Appendix Z.4). The long-document numbers of the submission used the default two passes and are therefore conservative.

Cost. Teacher targets take 21–34 s per 8-paper document and 64–72 s per 16-paper document (Table 33). An update takes 0.37 s on 8-paper documents at 2,048 entries, so the longer schedule adds a few minutes per document.

W3. Section 3 and the presentation. We followed your suggestions.

  • Section 3 is condensed to one paragraph; the formal statements and proofs are in Appendix T.
  • The space is used to explain Attention Matching (Section 2, Stage 1). For each KV head, AM selects a subset of the original keys and fits, on a set of reference queries, a bias β for each kept entry, so that the attention mass on the compact entries matches that on the full cache, and a value for each kept entry, so that the attention output matches. At read time the bias of an entry is added to its attention logit. NCC keeps the biases, the positions and the template entries fixed and trains the keys and values of the document entries.
  • The six-model table and the pure-attention result are in Section 5.2 of the main text (part 1).
  • The abstract and the introduction are rewritten around the setting, the method and the results, and state the scope; in the results, Sections 5.1 to 5.3 now open with their finding.

Nitpick: "rows"; 451 of 512. Thank you; both are changed. The revised PDF says "KV entries" throughout and states, where the budget is introduced, that of the 512 entries per KV head 451 hold document tokens and 61 hold the chat template.

We hope this answers your questions, and we would be glad to clarify anything further.

Reviewer_ZExEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Neural Context Compiler (NCC) tackles the problem of compiling (and compressing) the KV-cache of a document into a "compact" cache that an LLM can then query to retrieve information for particular prompts. This is a valuable problem, KV-cache sizes for modern LLMs are large and prohibit production efficiency, particularly in the long-context regime.

Interestingly, NCC achieves this through a pretty novel approach compared to SoTA. Typical KV-cache compression methods work on eviction (e.g. SnapKV). However, NCC proposes Attention-matching (AM), fitting a small set of key/value vectors to the original document, and then does a little bit of optimization on top of this, applying source-continuation distillation.

Critical strengths

The paper proposes two novel techniques, with very good results (table 1).

The paper is well motivated with Section 3 detailing precisely why the source-continuation distillation objective is necessary.

Critical weaknesses

My main point of contention is the use of single model: Qwen-3.6-27B. A KV-cache compression algorithm should be rigorously tested across model families, but more importantly: the particular Qwen model is a GDN hybrid, and in some ways goes against the main motivation of the text. A hybrid model itself has lower KV-cache consumption given 75% of the layers are linear-RNNs. The paper would do well to demonstrate the technique across model families, preferably models where full attention is used throughout the layers (e.g. LLama).

Questions for authors

Can you evaluate your method on different LLMs, primarily full-softmax only LLMs (e.g. LLama)?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZExE

Thank you for the review and for the clear question about model families. It led us to bring the paper's multi-model evidence into the main text and to state the scope of the method up front. A revised PDF is uploaded (changes in blue). Table numbers below are the submission's; "KV entries" are its "rows".

Evaluation across models. The submission evaluates the method on six models from three families (Appendices B and C). The table is now in Section 5.2 of the main text:

Model (recurrent layers)Full contextAMNCCNCC − AM, recallNCC − AM, QA F1
Qwen3.6-27B, main (Gated DeltaNet)99.962.379.8+17.5 [14.7, 20.5]+6.1 [2.4, 9.7]
Qwen3.5-9B (Gated DeltaNet)99.059.369.6+10.2 [7.7, 12.8]+2.8 [−1.6, 7.3]
Nemotron Nano 9B v2 (Mamba-2)67.029.639.3+9.7 [6.3, 13.3]−0.8 [−4.6, 2.5]
Nemotron 3 Nano 4B (Mamba-2)67.425.833.6+7.8 [5.2, 10.4]−0.4 [−4.4, 3.6]
Olmo Hybrid 7B (Gated DeltaNet)53.625.424.0−1.4 [−5.0, 2.1]−6.8 [−11.5, −2.3]
Qwen3-8B (none: pure attention)87.943.044.8+1.8 [−3.1, 6.7]−5.5 [−10.9, −0.1]

Full context, AM and NCC: constrained recall of masked numbers (%) on 64 papers at 512 KV entries per KV head; differences with 95% intervals. Attention Matching (AM; Zweiger et al., 2026) is the prior method from which NCC starts; NCC adds the refinement. The learning rate is set per model, and Appendices B and C of the revised PDF report every setting that was run.

Your question: full-attention models such as Llama. The recall gain over AM replicates on four of five hybrids, from two families and with two kinds of recurrent layer. On the one pure-attention model we tested, Qwen3-8B, refinement does not improve on AM, and AM alone is the better choice there. We did not run a Llama-family model. The revision therefore presents NCC as a method for hybrid models, and says so in the abstract, the introduction and a Limitations paragraph.

A hybrid already has a smaller KV cache. This is a fair point for short documents: on single papers the whole state shrinks 3.7× (Table 1), because the fixed recurrent state is most of what remains. Long documents are the case we target, and there the KV cache is the part that grows: 64 KiB per token on the main model, so that the full-context state of a 76K–97K-token document takes 5.3 GiB (Table 22). At that length the whole state shrinks 26.7× with 2,048 entries and 50.5× with 512 (Table 1), and 361 compiled 512-entry states fit on one GPU where 7 full caches fit (Table 23). The revised abstract and introduction are written around this setting.

The gain over AM is intact without the document's recurrent state. In a new experiment on the 64 papers with the main model (Appendix Z.1 of the revised PDF), we replaced the document's recurrent state at read time with that of an empty document. The gain of NCC over AM is intact: +20.3 [17.1, 23.6] points of constrained recall, against +17.8 [14.6, 21.0] with the full-document state in the same run. Entries refined beside the empty-document state and read with it reach 78.2% recall and 42.1 QA F1, against 79.4% and 45.9 in the paper's setting, read in the same run (differences −1.2 [−3.6, 1.1] and −3.7 [−8.0, 0.5]). The refined KV entries themselves carry the document.

Presentation. The abstract and the introduction are rewritten around the setting, the method and the results. The argument of Section 3, which you found useful, is kept in a condensed form: one paragraph in the main text, with the formal statements and proofs in Appendix T. The space is used to explain Attention Matching (Section 2).

We would be glad to answer further questions.

AuthorsOpenReview ↗

Gentle reminder for Reviewer ZExE

Dear Reviewer ZExE,

Thank you again for your review and for the time you have given to our submission. We understand that this is a busy time. If time permits before the discussion period closes, we would be grateful to know whether our reply above answers your question about other model families and your point that a hybrid model already has a smaller KV cache. The six-model table is now in Section 5.2 of the revised PDF, and the paper states up front that its scope is hybrid models. If anything remains open, we would be glad to clarify it.

Thank you for your time and consideration.

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the four reviewers for careful and constructive reviews. Your questions led us to run five new experiments during the discussion, and we think the paper is clearer for them. A revised PDF is uploaded with this response: changes are marked in blue, and the main text has 10 pages.

The method. NCC compiles a document, once, into a compact native KV cache for hybrid long-context models (attention layers interleaved with linear-recurrent layers). It starts from an Attention Matching (AM) cache and refines its keys and values by distilling the full-context model on the document's own text. The refinement needs no synthetic questions and decodes no text.

What the new experiments show (main model, Qwen3.6-27B; 95% intervals in brackets; details in the reviewer threads and in Appendix Z of the revised PDF)

  1. The refined KV entries themselves carry the document (P9m7, ZExE, EMpK; Z.1). Replacing the full-document recurrent state with that of an empty document, a truncated document or a different paper leaves the gain over AM intact: NCC − AM is +17.8 [14.6, 21.0] points of constrained number recall with the full state and +20.3 [17.1, 23.6] with the empty one.
  2. Against a stronger self-study control, NCC holds up, and its refinement generates nothing (P9m7; Z.4). The control now covers the whole long document with a 2–6× larger question bank. Given the same number of updates, NCC is ahead at 2,048 KV entries (recall +8.9 [5.1, 12.6] and +7.3 [3.0, 11.9] on 8- and 16-paper documents; QA F1 +4.5 [0.1, 8.8] and +7.8 [3.0, 12.9]) and not distinguishable at 512 entries (recall −0.4 [−4.0, 2.8] and −2.2 [−5.7, 1.7]). With, on average, no more GPU time than self-study's update steps alone (its generation counted as free), NCC is still ahead in recall on 8-paper documents at 2,048 entries (+5.8 [2.1, 9.6]). Self-study's numbers are those of its original run.
  3. A simple recipe scales with length (P9m7; Z.2). On 34K–51K-token documents at 2,048 entries, more passes over the same spans raise recall from 56.2% to 65.4% (+9.2 [6.5, 11.5]) and QA F1 from 38.7 to 43.0 (+4.3 [0.7, 7.5]; full context: 46.2). Recall follows the K/V compression (cached tokens per KV entry): for a new setting we predicted 41–44% recall before the run and measured 44.2%.
  4. Compilation is practical (EMpK; Z.3). Without any generated text, a 34K–51K-token document compiles in a median of 587 s and a 76K–97K-token document in 957 s (512 KV entries per KV head; 40% and 39% of the standard pipeline's time); compilation then pays for itself against per-query prefill after 121 and 89 queries.
  5. The method carries over to a new domain and task (UJa6; Z.5). On LongBench QMSum (query-focused summarization of 35 meeting transcripts, 200 queries, recipe unchanged), NCC scores 18.2 ROUGE-L against 12.4 for AM (+5.8 [4.4, 7.4]) and 17.5 for the full-context reader (+0.7 [−0.9, 2.4]), with a 7.3× smaller state.

Every run first re-reads or retrains the corresponding states of the submission: the re-reads are within 0.9 points of recall and 0.6 F1 of the printed values, and the retrained default arms within 1.6 points of recall.

Scope. The method is for hybrid models. With a learning rate set per model, the recall gain over AM replicates on four of five hybrids, from two families (+7.8 to +17.5 points, every interval above zero); on the one pure-attention model we tested (Qwen3-8B), refinement does not improve on AM, and AM alone is the better choice there.

Changes in the revised PDF

  • Abstract and introduction: rewritten around the setting (hybrid long-context models), the method and the results; they state the scope, as does a new Limitations paragraph (all reviewers).
  • Section 3: condensed to one paragraph, with the formal statements and proofs in Appendix T; Section 2 now explains Attention Matching: what is matched, and how the per-entry bias enters at read time (P9m7).
  • Section 5.2: the six-model table and the pure-attention result are in the main text (P9m7, ZExE, EMpK).
  • "Rows" is "KV entries" throughout; where the budget is introduced, the split of the 512 entries into 451 document and 61 template entries is stated (P9m7).
  • References: the template's single-column format (UJa6).
  • New appendices: X (results by kind of text) and Y (failure analysis) (UJa6); Z.1–Z.5 (the five experiments above).
  • Appendix J reports the full-coverage self-study control, Appendices B and C every learning-rate setting that was run, and Appendix F the deployment comparison (P9m7, EMpK).

We answer each reviewer's points in the reviewer threads and would be glad to continue the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.