acceptodds
Under review as a conference paper at ICLR 2027

Cache the Encoder Within: Compact, Reusable Memory across LLM Queries

Abstract

Repeated queries over shared documents incur redundant encoding, while caching model states introduces persistent storage costs. Building on CoMem's intermediate-state interface, EncBank treats a pretrained LLM's lower layers as a reusable document encoder and compactly stores their outputs for an adapted upper-layer reader. A self-distilled suffix adapter is shared across storage precisions within each backbone, without quantization-specific retraining. Across five benchmark suites on three Qwen backbones spanning different sizes and full-attention and hybrid architectures, 4-bit storage keeps each reported benchmark aggregate within one score point of native-precision EncBank. In a fixed Qwen3-8B workload, it retains 28.1% of the native-precision persistent GPU store. Separate native-precision controls yield a selected-pack prefill speedup over same-evidence, same-adapter text replay, at a 3.12-point RULER accuracy cost. A native-precision Qwen3.8-27B configuration also passes 70 of 89 Terminal-Bench 2.1 tasks. EncBank thus combines reusable computation with compact memory, while task fidelity and end-to-end benefits remain dependent on the workload, preparation costs, and reuse frequency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, persistent memory, intermediate representations, representation reuse, memory quantization, efficient inference, retrieval-augmented generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_cQrsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

EncBank extends CoMem, which proposes to store the activation of document chunks until a certain layer, enabling order-independent chunk reuse, with fine-tuned upper layers still doing order-sensitive attention. EncBank mainly extends CoMem by showing such stored activations can be quantized.

Critical strengths
  • Order-indepent chunk reuse is highly relavent to RAG systems and are relevant to modern LLM usage.
Critical weaknesses
  • Limited novelty over CoMem While novelty is a tricky weakness to address and I often try my best not to bring it up, it seems like this work really is mostly applying typical group-wise quantization to CoMem. The fact that this can work is, in my opinion, very expected, as there already exists a ton of KV Cache quantization work, which is highly relevant to activation quantization. There even exist works like XQuant showing that activation quantization is very possible. While their inference pipeline is different from EncBank, it seems fairly trivial to conclude that the activations stored by CoMem can withstand some quantization pressure, especially when the upper layers are open for fine-tuning.

  • Unfair comparison in Table 1 I was very surprised when reading Table 1 and seeing that the proposed EncBank is able to beat Qwen3-8B by such a huge margin. But then I realized these baselines are seeing the full source as context, where the proposed EncBank only sees retrieved sources. This comparison is fundamentally confounded because the BM25-based retriever is independent of EncBank and could also be applied to Dense. This setting renders Table 1 largely uninformative beyond showing that CoMem can be quantized to different budgets.

  • Thin and dated comparison with chunk-reuse methods. One other aspect of this work is to compare with CacheBlend-like work on chunk reuse. This is mostly done in Table 4, using just H16 (which is essentially the CoMem-style setting if my understanding is correct) and only evaluated on two datasets. CacheBlend is an early work in this category, but none of the stronger follow-up methods from the past few years are compared. The same criticism can also be extended to KIVI which is also dated, though I view this as a more minor issue since KV Cache quantization is probably not the most relevant line of work for EncBank.

  • Limited benefit of H4/H8 beyond storage Table 5 seems to suggest that the only major benefit of H4/H8 over H16 is storage. The request-time performance barely improves. More concerningly, EncBank is actually slower than Dense during decoding, even though algorithmically it should not be. This suggests substantial implementation overhead and makes the systems results hard to interpret.

It is possible that I have a major misunderstanding of this work’s positioning, and I am open to changing my mind if that is the case. Based on my current reading, however, the main contribution appears to be applying standard group-wise quantization to CoMem’s stored activations. I do not think combining existing ideas is inherently problematic, but such a combination should usually be technically nontrivial or meaningfully underexplored. Given the extensive literature on KV Cache and activation quantization, I currently do not see that here.

On the empirical evidence side, I feel like the most relevant setting is some kind of RAG evaluation, similar to CacheBlend, where retrieved passages are fixed and different chunk-reuse methods are compared directly. I would simply like to see broader coverage in terms of standard RAG benchmarks, baseline methods, and model backbones.

Questions for authors
  • What is the difference between EncBank-H16 and CoMem?
  • Why is KIVI slower than Dense in Table 5? I thought the common understanding is that LLM decoding is often memory-bound, in which case quantization should help.
  • Since EncBank fine-tune the upper layers of the LLM, would it effect LLM's performance on typical knowledge and reasoning tasks?
  • EncBank constructs chunks by their source order, but many sources are naturally orderless. How will EncBank perform under such tasks? Also is there any ablation on such ordering?
Reviewer_VTd7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper propose this fairly simple and straightforward method called EncBank, which processes each document chunk through the lower layers of an LLM once and saves the resulting hidden states.

For every new query, it first retrieves relevant chunks using BM25, decompresses their stored hidden states, and runs only the upper model layers over those states and the query. A lora adapter are used for the upper layers to handle independently encoded chunks.

From my reading, the main addition over CoMem is storing these states in 8-bit or 4-bit form.

Critical strengths
  • Given repeated-query workload, encbank is a reasonable solution to speed up and save inference cost

  • Good profiling on the storage, preparation, prefill, decoding, and amortization costs.

  • Evaluations are fairly comprehensive, with different control ladder like H16/H8/H4.

Critical weaknesses
  • The reusable intermediate-state interface and suffix adaptation come directly from CoMem. To me it seems like the only addition is the 4bit quantization that comes later to reduce storage cost.

  • Several baselines are a little questionable. For example, the large rulder advantage over Dense partly results from Dense exceeding its unextended context window. (correct me if i am wrong) The chunk-KV control uses an adapter trained for EncBank and a custom synchronous implementation.

  • Another one i found that is related to the main task, in the results authors show encbank improve over dense on tb by 9 tasks. However, this is not strong evidence that encbank caused the improvement because the two runs used different adapters, serving backends, and execution paths, with only one retained outcome per task. The paper appropriately calls it descriptive rather than a controlled comparison, which is a little concerning.

Questions for authors
  1. Maybe the author can help me understand more about what is technically novel beyond applying ordinary 4-bit quantization to CoMem states?

  2. is there a reason why the dense, kv-cache, and encbank baselines use different retrieval and serving backends?

  3. it would also be useful for the authors to discuss more about what workloads have enough repeated queries to recover preparation and storage costs.

Reviewer_RTfJOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Repeated queries over the same documents make a language model encode the same context again and again. The paper builds on CoMem, which stores one hidden state per token at a chosen layer boundary and continues through the remaining layers. EncBank treats the lower layers as a reusable document encoder and stores the states in 4, 8, or 16 bit form. It retrieves chunks with token level BM25 and reads the reconstructed states with a small self distilled adapter shared across all precisions. Evaluation covers three Qwen backbones and five long context benchmarks, with matched controls for the speed and fidelity cost of reuse.

Critical strengths
  1. The fidelity cost of reuse is measured under a matched control. Adapter, evidence, order, and decoding are fixed while only the reuse path changes, and a continuous prefix oracle recovers replay exactly. The reported cost is an attribution, not a side effect of a different pipeline.
Critical weaknesses
  1. The compute reuse in the title nearly washes out. The 1.40x figure is prefill only, and adding decode brings it to 1.07 to 1.09. The matched arm that charges preparation has EncBank slower than same adapter replay, 7.012 s against 6.400 s in Table 4b. What survives is a smaller store and a workload dependent payback.

  2. Half of the quality evidence cannot be checked. The 9B and 27B rows have no checkpoint identity, per item predictions, or judge binding. The Terminal Bench comparison runs the two methods on different adapters and backbones with one outcome per task. The paper itself marks these rows as descriptive.

  3. The close aggregate result holds only for whole benchmarks. Per length swings on one benchmark reach six points up and four down, and one dataset has a confidence interval that excludes zero. The paper calls the observation descriptive rather than a non inferiority test, yet the abstract states it as a property.

Questions for authors
  1. The adapted residual path beats adapted chunk KV but loses to unadapted chunk KV on LongEval, while on Qasper it beats both. Can you train one adapter on the chunk KV path and report the same grid, so the comparison is not confounded by the shared adapter.

  2. In the equal latency study the deficit tracks the retriever. Does selection with a stronger retriever combined with residual reading close the gap, or is the residual representation itself the problem.

  3. Can you release per task verifier outputs for the Terminal Bench runs, and repeat a fixed subset of tasks several times under the same serving path.

  4. Why is the 4 bit configuration missing from the amortization grid. One measured row would give the deployed setting its payback count.

  5. What sample size would a non inferiority test at a one point margin need on the five aggregates, and do the stored per item predictions already support it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.