- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposed a cache-compatible context distillation framework, which is used to manage a set of distilled document memories. Each document is distilled into one independent LoRA adapter. The adapters are trained to generate from the query prefix KV cache, which is computed by the frozen base model, so only one shared cache is needed for both candidate evaluation and fallback to base model. For memory selection, the method combine external retrieval with an internal router, and the router check the first token of each candidate.
Critical strengths
In my view, the most valuable part is the problem formulation. The authors formulate the deployment of many distilled memories as two decisions: which memory to select, and whether to activate it. Previous works mostly focus on how to write a single memory, and this kind of framing is largely missing in them. Also, the recursive interpretation in the appendix is a good critique of the cumulative paradigm. It shows that, in the ideal limit, a shared updated state is just the base model conditioned on all documents concatenated together. So it also have the long-context degradation problem, and there is no way to access one memory separately.
Another strength is the cache-compatible design, which solve several problems with one choice. Because the adapters are trained on the query prefix from frozen base model, the interface of training and inference is consistent. All candidates can share one prefix cache, so routing only need one prefill and a few one-token probes, and the fallback branch can also reuse the same cache. The comparison between different training interfaces show that this consistency alone bring several points of improvement. The experimental setup is also careful.
Critical weaknesses
The statistical reporting is not enough in my opinion. For all experiments, there is no information about variance or random seeds, and no table report confidence intervals. Both the adapter training and the sampling of synthetic questions are stochastic, so the results can change between runs. Since the main gaps in the results are big, I am not worry about the direction of the conclusions. But for the smaller comparisons, for example routing with three candidates vs. five candidates, or the drop of routing accuracy when the memory bank become larger, it is hard to interpret them without variance.
The evidence on scale is also limited, especially for a paper about deploying many memories. The largest backbone is only 7B, and the largest memory bank only have 355 adapters. Even at this size, routing accuracy already drop several points when the bank grows from 50 to 355 adapters. The external retrieval part is also only tested in this small scale, so we don't know if the candidate set can still include the correct adapter when the corpus has thousands or millions of documents. My concern is, the deployment scenario need a lot of memories to amortize the construction cost, but this is exactly the setting which is not evaluated.
Questions for authors
-
When the bank grows from 50 to 355 adapters, the top-1 routing accuracy drop from 69.49% to 65.07%, and the top-5 accuracy drop from 78.64% to 69.27%. What will happen if the bank is even larger? And if the corpus become thousands times larger, can the external retrieval still put the correct adapter into the candidate set?
-
When does a larger candidate set stop to help? Is the router more easy to select a wrong adapter if there are more candidates?
-
On SQuAD, the distilled memories only reach about half of the in-context upper bound, but on NarrativeQA they are much closer. What is the reason of this difference between tasks? In which deployment cases is the distilled memory good enough, compared to just putting the document into the prompt?
-
The gating threshold is calibrated on a mixture of training queries from NarrativeQA and CommonsenseQA. If the query mixture in real deployment is very different from this, how sensitive the gate will be?
3 comments
Response for Reviewer TqJW(2/2)
Q4:The gating threshold is calibrated on a mixture of training queries from NarrativeQA and CommonsenseQA. If the query mixture in real deployment is very different from this, how sensitive the gate will be?
A4. We expect a single fixed threshold to be sensitive to substantial shifts in the query distribution, particularly in real deployments where the usefulness of distilled memories varies across query types.
However, we can reframe this as a local calibration problem by periodically re-calibrating the threshold on a rolling window of recent deployment queries, using feedback on whether activating a memory improved the answer. This follows the same calibration principle as in the paper. The gate still uses first-token entropy to decide whether to activate a memory or fall back to the base model. The threshold is dynamically calibrated on recent deployment QA so that it can adapt as the query mixture changes.
I would like to thank the authors to address my previous comments and questions. I understand the current constrains on training budget and the authors will leave the evaluation of routing at large scale in the future work. The referenced technical report shows some evidence to support but I would like to see stronger evidence on the scalability of the work. All other comments are well explained and addressed. The trade-off evaluation on the routing accuracy proves when a larger candidate set stops work. Overall I am satisfied with the authors' response.
Response for Reviewer TqJW(1/2)
We thank the reviewer for their careful reading and constructive feedback. We respond to the reviewer’s questions below and hope our responses address the concerns
Q1: When the bank grows from 50 to 355 adapters, the top-1 routing accuracy drop from 69.49% to 65.07%, and the top-5 accuracy drop from 78.64% to 69.27%. What will happen if the bank is even larger? And if the corpus become thousands times larger, can the external retrieval still put the correct adapter into the candidate set?
A1: Larger adapter banks. When the NarrativeQA bank grows from 200 to 355 adapters, the router’s accuracy conditional on retrieving the correct adapter decreases only modestly, from 88.97% to 87.61%. At 355 adapters, Top-5 routing still improves over top-1 retrieval by 4.20 points. This suggests that the router retains most of its ability to select the correct adapter once it is in the candidate set over this range. For larger banks, maintaining retrieval recall will be important to preserving routing accuracy. Therefore, it leads to the second quesstion: if the corpus become thousands times larger, can the external retrieval still put the correct adapter into the candidate set?
Larger retrieval corpus. Our first-stage retriever, Qwen3-Embedding-0.6B, has been evaluated on benchmarks with substantially larger corpora in the technical report[1].
These results characterize ranking quality on large retrieval benchmarks. The results provide evidence that, even with a thousands times larger corpus, the external retriever still have the capability to narrow the bank and give a candiadate sets at scales. Given our compute budget, we leave distilling ~100,000 adapters and evaluating routing at that scale to future work.
[1] Zhang Y, Li M, Long D, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models[J]. arXiv preprint arXiv:2506.05176, 2025.
Q2:When does a larger candidate set stop to help? Is the router more easy to select a wrong adapter if there are more candidates?
A2: We thanks the reviewer for the valuable question. As the candiate sets be larger, the recall of external retriver will be higher but it will be harder for the internal router to select the correct adapter. To measure this trade-off, we varied the candidate set size from 1 to 20 on NarrativeQA with a fixed bank of 355 adapters.
On NarrativeQA, a larger candidate set stop to help at around 10 candidates in our 355-adapter bank. Increasing candidate sets from 10 to 20 retrieved the correct adapter for 89 additional queries, but the overall accuracy change was −0.10 percentage points. Thus, a wider candidate set improved retrieval recall, while the additional candidates yielded little gain in final routing accuracy.
Q3:On SQuAD, the distilled memories only reach about half of the in-context upper bound, but on NarrativeQA they are much closer. What is the reason of this difference between tasks? In which deployment cases is the distilled memory good enough, compared to just putting the document into the prompt?
A3: We thank reviewer for suggesting us to revisit the Qwen2.5-7B results on SQuAD. We identified an incorrect LoRA rank setting in the original SQuAD run and updated the results using the intended rank of 64, and it won't change the paper’s main conclusions. We rebuilt the 300-adapter bank with the rank of 64 and evaluated it on all 2,642 SQuAD questions. We have updated Table 1 with the corrected results below. The corrected Ours@5 result improves by 19.29 F1 points and reaches about 77% of the in-context F1.