acceptodds
Under review as a conference paper at ICLR 2027

From Context Distillation to Latent Memory Management: Selecting and Activating Distilled Memories

Abstract

Context distillation internalizes a document into model parameters, allowing subsequent queries to be answered without repeatedly including the document in the prompt. While existing work primarily studies how to construct a memory from a single document, deploying a collection of such memories introduces two additional decisions: which memory is relevant to a query and whether the selected memory should be activated at all. We formulate these decisions as latent memory management with memory selection and memory activation, and introduce cache-compatible context distillation, which represents documents as independent LoRA adapters trained to operate on a query prefix KV cache produced by the frozen base model. This shared cache supports latent memory management without recomputing the query prefix, enabling an efficient system in which external retrieval produces a candidate set, internal routing selects one candidate, and Self-Gating decides whether to activate the selected memory. Across NarrativeQA, SQuAD, and LongBench, routing consistently improves over top-1 retrieval, while Self-Gating substantially recovers performance on context-agnostic queries with only small changes to document-specific QA. The same selection strategy also improves latent-token memories, suggesting that the management problem extends beyond LoRA-based representations. Our caching pipeline further reduces memory-construction time by up to 8.4×, peak memory by 4.1×, and training FLOPs by 21.9×, while adding little online management overhead. Overall, our results show that scaling context distillation requires not only writing memories, but also selecting when and which memories to use.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Latent Memory, Context Distillation, Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TqJWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed a cache-compatible context distillation framework, which is used to manage a set of distilled document memories. Each document is distilled into one independent LoRA adapter. The adapters are trained to generate from the query prefix KV cache, which is computed by the frozen base model, so only one shared cache is needed for both candidate evaluation and fallback to base model. For memory selection, the method combine external retrieval with an internal router, and the router check the first token of each candidate.

Critical strengths

In my view, the most valuable part is the problem formulation. The authors formulate the deployment of many distilled memories as two decisions: which memory to select, and whether to activate it. Previous works mostly focus on how to write a single memory, and this kind of framing is largely missing in them. Also, the recursive interpretation in the appendix is a good critique of the cumulative paradigm. It shows that, in the ideal limit, a shared updated state is just the base model conditioned on all documents concatenated together. So it also have the long-context degradation problem, and there is no way to access one memory separately.

Another strength is the cache-compatible design, which solve several problems with one choice. Because the adapters are trained on the query prefix from frozen base model, the interface of training and inference is consistent. All candidates can share one prefix cache, so routing only need one prefill and a few one-token probes, and the fallback branch can also reuse the same cache. The comparison between different training interfaces show that this consistency alone bring several points of improvement. The experimental setup is also careful.

Critical weaknesses

The statistical reporting is not enough in my opinion. For all experiments, there is no information about variance or random seeds, and no table report confidence intervals. Both the adapter training and the sampling of synthetic questions are stochastic, so the results can change between runs. Since the main gaps in the results are big, I am not worry about the direction of the conclusions. But for the smaller comparisons, for example routing with three candidates vs. five candidates, or the drop of routing accuracy when the memory bank become larger, it is hard to interpret them without variance.

The evidence on scale is also limited, especially for a paper about deploying many memories. The largest backbone is only 7B, and the largest memory bank only have 355 adapters. Even at this size, routing accuracy already drop several points when the bank grows from 50 to 355 adapters. The external retrieval part is also only tested in this small scale, so we don't know if the candidate set can still include the correct adapter when the corpus has thousands or millions of documents. My concern is, the deployment scenario need a lot of memories to amortize the construction cost, but this is exactly the setting which is not evaluated.

Questions for authors
  1. When the bank grows from 50 to 355 adapters, the top-1 routing accuracy drop from 69.49% to 65.07%, and the top-5 accuracy drop from 78.64% to 69.27%. What will happen if the bank is even larger? And if the corpus become thousands times larger, can the external retrieval still put the correct adapter into the candidate set?

  2. When does a larger candidate set stop to help? Is the router more easy to select a wrong adapter if there are more candidates?

  3. On SQuAD, the distilled memories only reach about half of the in-context upper bound, but on NarrativeQA they are much closer. What is the reason of this difference between tasks? In which deployment cases is the distilled memory good enough, compared to just putting the document into the prompt?

  4. The gating threshold is calibrated on a mixture of training queries from NarrativeQA and CommonsenseQA. If the query mixture in real deployment is very different from this, how sensitive the gate will be?

3 comments
AuthorsOpenReview ↗

Response for Reviewer TqJW(1/2)

We thank the reviewer for their careful reading and constructive feedback. We respond to the reviewer’s questions below and hope our responses address the concerns

Q1: When the bank grows from 50 to 355 adapters, the top-1 routing accuracy drop from 69.49% to 65.07%, and the top-5 accuracy drop from 78.64% to 69.27%. What will happen if the bank is even larger? And if the corpus become thousands times larger, can the external retrieval still put the correct adapter into the candidate set?

A1: Larger adapter banks. When the NarrativeQA bank grows from 200 to 355 adapters, the router’s accuracy conditional on retrieving the correct adapter decreases only modestly, from 88.97% to 87.61%. At 355 adapters, Top-5 routing still improves over top-1 retrieval by 4.20 points. This suggests that the router retains most of its ability to select the correct adapter once it is in the candidate set over this range. For larger banks, maintaining retrieval recall will be important to preserving routing accuracy. Therefore, it leads to the second quesstion: if the corpus become thousands times larger, can the external retrieval still put the correct adapter into the candidate set?

Larger retrieval corpus. Our first-stage retriever, Qwen3-Embedding-0.6B, has been evaluated on benchmarks with substantially larger corpora in the technical report[1].

DatasetCorpus sizenDCG@10
MLDR200K50.26
CoIR CodeSearchNet~1M84.69
CodeFeedback-ST~156K86.39
Synthetic Text2SQL~106K76.74

These results characterize ranking quality on large retrieval benchmarks. The results provide evidence that, even with a thousands times larger corpus, the external retriever still have the capability to narrow the bank and give a candiadate sets at scales. Given our compute budget, we leave distilling ~100,000 adapters and evaluating routing at that scale to future work.

[1] Zhang Y, Li M, Long D, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models[J]. arXiv preprint arXiv:2506.05176, 2025.

Q2:When does a larger candidate set stop to help? Is the router more easy to select a wrong adapter if there are more candidates?

A2: We thanks the reviewer for the valuable question. As the candiate sets be larger, the recall of external retriver will be higher but it will be harder for the internal router to select the correct adapter. To measure this trade-off, we varied the candidate set size from 1 to 20 on NarrativeQA with a fixed bank of 355 adapters.

On NarrativeQA, a larger candidate set stop to help at around 10 candidates in our 355-adapter bank. Increasing candidate sets from 10 to 20 retrieved the correct adapter for 89 additional queries, but the overall accuracy change was −0.10 percentage points. Thus, a wider candidate set improved retrieval recall, while the additional candidates yielded little gain in final routing accuracy.

CandidatesRecallRouting accuracyAccuracy given a retrieval hit
164.75%64.75%100.00%
375.15%65.65%87.36%
578.90%66.45%84.22%
1083.65%66.95%80.04%
2088.10%66.85%75.88%

Q3:On SQuAD, the distilled memories only reach about half of the in-context upper bound, but on NarrativeQA they are much closer. What is the reason of this difference between tasks? In which deployment cases is the distilled memory good enough, compared to just putting the document into the prompt?

A3: We thank reviewer for suggesting us to revisit the Qwen2.5-7B results on SQuAD. We identified an incorrect LoRA rank setting in the original SQuAD run and updated the results using the intended rank of 64, and it won't change the paper’s main conclusions. We rebuilt the 300-adapter bank with the rank of 64 and evaluated it on all 2,642 SQuAD questions. We have updated Table 1 with the corrected results below. The corrected Ours@5 result improves by 19.29 F1 points and reaches about 77% of the in-context F1.

MethodOriginal EMOriginal F1Updated EMUpdated F1
In-context (gold document)77.5584.8877.5584.88
Ours@134.8544.9451.8261.18
Ours@336.3446.5755.6065.33
Ours@536.3046.4555.9065.74
AuthorsOpenReview ↗

Response for Reviewer TqJW(2/2)

Q4:The gating threshold is calibrated on a mixture of training queries from NarrativeQA and CommonsenseQA. If the query mixture in real deployment is very different from this, how sensitive the gate will be?

A4. We expect a single fixed threshold to be sensitive to substantial shifts in the query distribution, particularly in real deployments where the usefulness of distilled memories varies across query types.

However, we can reframe this as a local calibration problem by periodically re-calibrating the threshold on a rolling window of recent deployment queries, using feedback on whether activating a memory improved the answer. This follows the same calibration principle as in the paper. The gate still uses first-token entropy to decide whether to activate a memory or fall back to the base model. The threshold is dynamically calibrated on recent deployment QA so that it can adapt as the query mixture changes.

I would like to thank the authors to address my previous comments and questions. I understand the current constrains on training budget and the authors will leave the evaluation of routing at large scale in the future work. The referenced technical report shows some evidence to support but I would like to see stronger evidence on the scalability of the work. All other comments are well explained and addressed. The trade-off evaluation on the routing accuracy proves when a larger candidate set stops work. Overall I am satisfied with the authors' response.

Reviewer_Ty9BOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper formulates the task of managing KV memories and proposes a solution with compatible context distillation and inference processes. The method is validated on Qwen and Llama models using QA datasets, showing consistent gains over context-compression and context-distillation baselines.

Critical strengths
  1. The problem studied, latent memory management, is practical and relevant to recent literature.
  2. There is performance and efficiency advantage compared to baselines.
  3. Research questions are clearly posed.
  4. The paper is rigorous in terms of formulation, ablations, justification of components, and extensions to other forms of memories.
  5. I appreciate the detailed AI disclosure.
Critical weaknesses
  1. Since the evaluation is done only on Llama 3.2 1B and Qwen2.5 {0.5B, 7B}, we do not know whether results will generalize to larger or non-Llama, non-Qwen models.
  2. There is no clear argument as to why previous methods or simple adaptations of them would be worse than your pipeline. The related work section just states how your focus differs from the literature.
  3. This is minor, but base model and the in-context baseline might not necessarily be the lowerbound or upperbound; an example is that a method might be too bad that it corrupts performance to below the base model; sometimes in-context memory might not be the best either since models could degrade at very long context lengths.
Questions for authors
  1. In "Why first-token entropy?", the authors justified the advantage of the first-token entropy over dataset identity or query format as a separation signal. Could the authors propose any alternative signal and argue, conceptually or empirically, as to whether such signal(s) can be better or worse than first-token entropy?
  2. Could the authors conduct an additional experiment on some models larger than 8B parameters? If your compute is limited, please provide your thoughts on whether or not your method can scale to larger models.
2 comments

We thank the reviewer for their careful reading and constructive feedback. We respond to the reviewer’s questions below and hope our responses address the concerns

Q1: In "Why first-token entropy?", the authors justified the advantage of the first-token entropy over dataset identity or query format as a separation signal. Could the authors propose any alternative signal and argue, conceptually or empirically, as to whether such signal(s) can be better or worse than first-token entropy?

A1: First-token entropy provides an early signal of the model’s confidence when using the selected adapter. The gate can decide whether to keep the adapter or fall back to the base model after a single decoding step. If the router has already computed the first-token distribution, the gate can reuse it.

We compare first-token entropy with two alternatives on Qwen2.5-0.5B:

  • Sum. We sum the entropies of the first kk generated tokens and activate the adapter if the sum is below a calibrated threshold.
  • Vector. We keep the first kk entropies as separate features, fit a logistic regression classifier, and apply a calibrated threshold to its score.

Summing more token entropies lowers balanced accuracy. For Vector setting, the two-token vector improves balanced accuracy by 1.34 percentage points over first-token entropy at approximately twice the probe FLOPs. Using four tokens adds only 0.07 points over two tokens and doubles the probe FLOPs again. This trade-off motivates using first-token entropy for gating.

TokensSum balanced accuracyVector balanced accuracyProbe GFLOPs
170.16%70.16%1.06
269.60%71.51%2.12
468.04%71.58%4.24
865.99%71.42%8.49

Q2: Could the authors conduct an additional experiment on some models larger than 8B parameters? If your compute is limited, please provide your thoughts on whether or not your method can scale to larger models.

A2. We evaluated routing and self-gating on Qwen2.5-14B. Due to the compute limitation, we used rank-16 LoRA adapters instead of rank 64 used in our main experiments. The lower rank limits the capacity of the distilled memories. However, the purpose of this experiment is to assess whether routing and self-gating remain effective on a 14B backbone, so it won't affect our conclusion.

Table 1:Routing and self-gating on Qwen2.5-14B

MethodNarrativeQA ROUGE-1NarrativeQA ROUGE-LCommonsenseQA Accuracy
Base Model19.3218.7686.73%
Ours@126.0825.1486.08%
Ours@327.1426.2086.49%
Ours@528.3627.1985.91%

On NarrativeQA, routing over three or five candidates improves both ROUGE scores over top-1 selection. Ours@5 gains 2.05 ROUGE-L points over Ours@1. With self-gating enabled, all three settings maintain most of the base model’s CommonsenseQA accuracy. These results show that routing improves document-specific QA on a 14B backbone while the gated system largely preserves accuracy on context-agnostic queries.

Thank you for providing the clarifications. Overall I am satisfied with the response and updated my score accordingly.

I hope the authors can conduct more systematic experiments (e.g., considering more types of signals, fixing your wrong implementations, and scaling to more models and data) and include explanation to address my and other reviewers' concerns in the camera-ready version. For scaling in particular, if the method fails to generalize to much larger models or some particular datasets, I think diagnosing why and proposing grounded future directions with some preliminary experiments could also be helpful enough.

I will update here with more suggestions if I can think of any during the rest of the rebuttal period. In general I think the paper (or a potential camera-ready version of it) meets conceptual standards for ICLR, though there could be some empirical improvements.

Reviewer_HqtZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how a language model can use a library of document memories stored as LoRA adapters. Each adapter is trained to retain information from one document. A question about a particular story should use that story's adapter, while a general question may be better answered by the base model. The challenge is to select the relevant memory and decide whether activating it will help.

The main contribution lets candidate adapters share the computation for most of the query prompt. The frozen base model processes this prefix, and each adapter takes over at the last prompt token. A router combines retrieval scores with adapter predictions to select a memory, and an entropy threshold decides whether to use it or fall back to the base model. Experiments across three backbones show generally modest routing gains and improved CommonsenseQA performance with fallback. Cache reuse reduces costs relative to an uncached implementation, but the evaluated workloads do not recover the initial memory-construction cost under the paper's own analysis.

Critical strengths
  1. The base model processes most of the query, and each candidate adapter takes over at the last prompt token. Matching this interface during training and inference allows adapters to share expensive computation without making their predictions identical.

  2. Routing among three candidates improves SQuAD F1 over top-1 retrieval by 1.85, 1.63, and 1.20 points. Gating recovers 2.30 to 8.32 CommonsenseQA accuracy points while losing at most 1.26 NarrativeQA ROUGE-1 points. These results show useful improvements within the tested setup, and the ablations help separate the effects of routing, gating, and cache-compatible training.

Critical weaknesses
  1. The practical advantage over cached-text retrieval is not established. Figure 7 implies break-even at roughly 2,000 queries per document at NarrativeQA lengths, while the benchmark supplies about 30. The construction speedups compare cached with uncached distillation, rather than accounting for whether building adapters is worthwhile. Storage further complicates the comparison. Under the stated rank-64 configuration, a Qwen2.5-7B adapter is approximately 323 MB in bf16, compared with roughly 42 MB for the document KV cache at the average NarrativeQA length. A matched document-prefix-caching baseline is missing. This matters because the evidence does not yet establish a favorable end-to-end cost and accuracy tradeoff for the proposed memory system.

  2. The management evaluation does not isolate whether the selected memory is useful. The gate labels all NarrativeQA questions for activation and all CommonsenseQA questions for fallback. A NarrativeQA question paired with an incorrect memory still receives the activation label. The experiment can therefore reward recognition of dataset-specific question styles without testing whether activation improves the answer. The router also lacks a comparison with a text/query reranker over identical candidates. These omissions limit the central claim that the management layer selects and activates beneficial memories, beyond demonstrating gains against the particular baselines used.

Questions for authors
  1. Are the approximate break-even and storage estimates above consistent with your implementation? What document-reuse rate supports the practical efficiency claim once construction, adapter loading, and storage are included?

  2. Can you compare against retrieved-text inference with document-prefix caching and a text/query reranker using identical candidates? Reporting answer quality alongside construction, loading, and per-query costs would directly affect my assessment of the system's practical value. This would require additional evaluation.

  3. For the same questions within one dataset, does the gate distinguish helpful from incorrect or absent memories when judged by improvement over the base model's answer? Testing this with fixed calibration thresholds would directly address the dataset-membership confound. This would require an additional experiment.

7 comments
AuthorsOpenReview ↗

Response for Reviewer HqtZ(1/4)

We thank the reviewer for their careful reading and constructive feedback. We respond to the reviewer’s questions below and hope our responses address the concerns

Q1: Are the approximate break-even and storage estimates above consistent with your implementation? What document-reuse rate supports the practical efficiency claim once construction, adapter loading, and storage are included?

A1: We agree that when and whether to distill a document(compared with document cache), is a fundamental question for context distillation. The answer depends on the trade-off between answer quality, construction cost, storage, and per-query cost. But we also clarify that our contribution is to make distilled document memories cheaper to construct and manage, which is build on the existing adapter-based memory formulation. We introduce cache-compatible distillation, which allows the same query-prefix cache to be reused when evaluating candidate adapters and falling back to the base model.

To address the reviewer’s question about efficency, in A1, we compare these memories with document KV-cache under the paper’s Qwen2.5-0.5B and rank-64 LoRA setting, and we further compare the performance in A2.

1. Memory Storage. As the reviewer estimated, the document KV cache requires less storage for short contexts. Its size grows with document length, whereas the adapter size is fixed. At 8K and 16K tokens, the adapter uses less storage than the document KV cache.

Table 1: Additional Memory Storage (MB)

Setting5001K2K4K8K16K
Ours70.470.470.470.470.470.4
Text w/o Cache000000
Text with Cache6.112.324.649.298.3196.6

2. Memory Construction Cost. In Table 2, we report the memory construction cost. Cache-compatible distillation substantially reduces construction FLOPs. Across all evaluated document lengths, constructing an adapter still requires more computation than building a document KV-cache.

Table 2: Memory Construction FLOPs (TFLOPs per document)

Setting5001K2K4K8K16K
Ours1256.11284.01339.91451.91677.12131.5
w/o Cache Compatibility5229.99158.417660.737245.786737.6227009.0
Text w/o Cache000000
Text with Cache0.370.761.603.558.4822.46

3. Inference Cost.

Let LL, QQ, and AA denote the context, query, and answer lengths. We compare the complexity among each setting. Reusing a document KV cache skips document prefill, but the query and answer tokens still attend to the cached document. This adds an O(L(Q+A))O(L(Q+A)) attention term. Our method generates from the query and adapter, so its inference FLOPs are independent of the original document length.

SettingPrefill attentionAnswer generation attentionTotal complexity
OursO(Q2)O(Q^2)O(A(Q+A))O(A(Q+A))O((Q+A)2)O((Q+A)^2)
Text w/o CacheO((L+Q)2)O((L+Q)^2)O(A(L+Q+A))O(A(L+Q+A))O((L+Q+A)2)O((L+Q+A)^2)
Text with CacheO(Q(L+Q))O(Q(L+Q))O(A(L+Q+A))O(A(L+Q+A))O((Q+A)2+L(Q+A))O((Q+A)^2+L(Q+A))

Table 3 reports inference FLOP per query. As predicted by the O(L(Q+A))O(L(Q+A)) attention term, text with cached inference rises from 40.11 to 102.77 GFLOPs as document length grows from 500 to 16K tokens, while our cost stays at 39.22 GFLOPs. Uncached text grows faster because it repeats document prefill, which includes quadratic attention.

Table 3: Inference FLOPs (GFLOPs per query)

Setting5001K2K4K8K16K
Ours39.2239.2239.2239.2239.2239.22
Text w/o Cache408.71800.841649.603605.178548.5122563.96
Text with Cache40.1142.1346.1754.2670.43102.77

4. Break-even The break-even is the number of queries per document needed for inference savings to cover the additional construction cost. It decreases with document length and is substantially higher against cached text than against uncached text.

Table 4: FLOP Break-even (queries per document)

Comparison5001K2K4K8K16K
Ours vs. Text w/o Cache3,3901,68483240819895
w/o Cache Compatibility vs. Text w/o Cache14,11212,00810,96010,44210,19210,078
Ours vs. Text with Cache1,410,933440,976192,56296,30053,46533,188

Overall, distilled memories can reduce storage and per-query FLOPs for long documents, while their higher construction cost requires substantial reuse to break even against document KV caching. Within the existing adapter-based memory formulation, cache-compatible distillation lowers construction costs and substantially reduces the number of break-even.

AuthorsOpenReview ↗

Response for Reviewer HqtZ(2/4)

Q2: Can you compare against retrieved-text inference with document-prefix caching and a text/query reranker using identical candidates? Reporting answer quality alongside construction, loading, and per-query costs would directly affect my assessment of the system's practical value. This would require additional evaluation.

A2: We compared retrieved-text inference, text reranking, and distilled-memory routing on identical candidate lists. We report answer quality on SQuAD and NarrativeQA and estimate per-query FLOPs.

1. Experiment Setting. For each query, RAG@K concatenates the top KK retrieved documents, Reranker@K uses Qwen3-Reranker-0.6B to select one document from the same candidates, and Ours@K selects a distilled adapter corresponding to one of those candidates.

We use Qwen3-Reranker-0.6B with default prompt: "Given a query, retrieve relevant passages that answer the query <Instruct>: {instruction}\n<Query>: {query}\n<Document>: {doc}" The standard reranker places the query before the document, making the document’s KV states query-dependent. We also tested a document-first prompt, which enable document-prefix caching but it reduced accuracy on both datasets (Table 1). We therefore use the standard query-first format for reranking in the remaining comparisons.

Table 1: Query-first v.s Document-first

DatasetCandidatesQuery-first ACC (%)Document-first ACC (%)
SQuADTop-382.9345.99
SQuADTop-585.2040.54
NarrativeQATop-374.7159.11
NarrativeQATop-577.7855.17

2. Efficiency Comparsion. Table 2a isolates candidate selection. Our router uses one first-token probe per candidate after a shared query prefill, so its probe cost is independent of document length. The query-first reranker scores the full text of each query–document pair. Table 2b includes both candidate selection and answer generation. Table 2a shows that our router’s management cost stays constant as documents grow. At 4K tokens per document, selecting from five candidates costs 5.32 GFLOPs with our 0.5B router, compared with 28,248.38 GFLOPs for the query-first text reranker. Table 2b shows that the savings extend to full inference. Including routing and answer generation, Ours@5 costs 43.47 GFLOPs per query, compared with 120.88 GFLOPs for RAG@5 with a warm document-prefix cache.

Table 2a: Management FLOPs with Document Length

Document Tokens5121,0244,09616,000
Top-3 0.5B Router3.193.193.193.19
Top-3 7B Router43.4743.4743.4743.47
Top-3 0.6B Reranker1,637.313,283.5316,949.03131,240.59
Top-5 0.5B Router5.325.325.325.32
Top-5 7B Router72.4572.4572.4572.45
Top-5 0.6B Reranker2,728.855,472.5628,248.38218,734.32

Table 2b. Inference FLOPs per Query on Qwen2.5-0.5B. We include Management cost in this table.

Method5121,0244,09616,000
Ours@139.2239.2239.2239.22
Ours@341.3441.3441.3441.34
Ours@543.4743.4743.4743.47
Reranker@3 with cache1,677.473,325.7617,003.68131,343.36
Reranker@5 with cache2,769.015,514.7928,303.03218,837.09
RAG@1 with cache40.1642.2354.65102.77
RAG@1 without cache417.87820.203,707.6922,563.96
RAG@3 with cache44.3050.5187.77232.14
RAG@3 without cache1,245.082,655.0015,376.23133,675.99
RAG@5 with cache48.4458.79120.88361.51
RAG@5 without cache2,162.484,850.5832,817.20332,868.40

For Qwen2.5-0.5B, the FLOPs for RAG@3 and RAG@5 at 16,000 tokens per document are analytical extrapolations since both prefixes exceed the model’s native 32K context window.

AuthorsOpenReview ↗

Response for Reviewer HqtZ(3/4)

3. Performance Comparsion. Both settings use the ground-truth document identity. Oracle ICL gives the base model the gold passage on SQuAD or the gold story summary on NarrativeQA. Oracle Adapter loads the adapter distilled from that document and gives the model only the question.

We report the performance in Table 3 and 4, where the RAG@k and Reranker@k use the text as input and ours@k use the distilled memory. The result shows that in most the the case, RAG@k and Reranker@k outperform our method. However, we argue that this gap mainly come from the memory formulation, as the Oracle ICL always outperform the Oracle Adapter with a large margin. We believe that with a good context distillation method which shows comparable performance with ICL, Ours@k can also achieve comparable perfomance with RAG@k and Reranker@k.

Table 3: F1 score on SQuAD We thank Reviewer TqJW for suggesting us to revisit the Qwen2.5-7B results on SQuAD. We identified an incorrect LoRA rank setting in the original SQuAD run and updated the results using the intended rank of 64, and it won't change the paper’s main conclusions.

MethodQwen2.5-0.5BQwen2.5-7BLlama-3.2-1B
Oracle ICL57.5184.8849.17
Oracle Adapter50.0575.4143.83
RAG@146.3067.7342.38
RAG@342.3169.8944.42
RAG@541.2471.3744.76
Reranker@146.3067.7342.38
Reranker@350.5972.8245.32
Reranker@551.7974.2746.13
Ours@141.2661.1836.27
Ours@343.1165.3337.47
Ours@543.2565.7437.74

Table 4: ROUGE-L on NarrativeQA

MethodQwen2.5-0.5BQwen2.5-7BLlama-3.2-1B
Oracle ICL39.5354.0941.08
Oracle Adapter30.2236.7831.41
RAG@127.9935.6129.22
RAG@320.7843.3828.19
RAG@516.8343.1127.43
Reranker@127.9935.6129.22
Reranker@331.0840.1232.47
Reranker@532.0241.6033.44
Ours@122.5429.2622.10
Ours@323.9531.3222.62
Ours@523.9431.7222.66

4. Combining the reranker with distilled memories.
We also incorporate the text reranker’s scores into our router. Both variants generate answers using the selected adapter. On SQuAD, adding our router improves selection accuracy and F1 across all six settings. On NarrativeQA, the reranker’s selection accuracy is already close to the ceiling set by candidate recall, leaving little room for improvement.

Table 5: Result of Reranker with Distilled Memory on SQuAD

BackboneCandidatesRecall (%)Reranker ACC (%)Reranker + Ours ACC (%)Reranker F1Reranker + Ours F1
Qwen2.5-0.5BTop-390.2082.9383.6545.9246.21
Qwen2.5-0.5BTop-594.2585.2085.8446.9347.22
Qwen2.5-7BTop-390.2082.9383.9967.0068.41
Qwen2.5-7BTop-594.2585.2086.6868.4769.66
Llama-3.2-1BTop-390.2082.9383.5039.8940.13
Llama-3.2-1BTop-594.2585.2085.2440.6840.85

Table 6. Reranking with Distilled Memory on NarrativeQA

BackboneCandidatesRecall (%)Reranker ACC (%)Reranker + Ours ACC (%)Reranker ROUGE-LReranker + Ours ROUGE-L
Qwen2.5-0.5BTop-375.6274.7174.6224.9925.03
Qwen2.5-0.5BTop-579.0777.7877.6425.7425.71
Qwen2.5-7BTop-375.6274.7174.7332.2432.31
Qwen2.5-7BTop-579.0777.7877.8133.2633.33
Llama-3.2-1BTop-375.6274.7174.4623.4323.38
Llama-3.2-1BTop-579.0777.7877.5423.9723.91

Overall, retrieved text generally yields higher answer quality, while routing over distilled memories keeps FLOPs independent of document length. The quality gap mainly stems from context distillation itself, since the gap between Oracle ICL and Oracle Adapter persists even when both methods use the gold document. The reranker integration further shows that our internal router can improve selection and answer quality within the distilled-memory setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.