- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
EncBank extends CoMem, which proposes to store the activation of document chunks until a certain layer, enabling order-independent chunk reuse, with fine-tuned upper layers still doing order-sensitive attention. EncBank mainly extends CoMem by showing such stored activations can be quantized.
Critical strengths
- Order-indepent chunk reuse is highly relavent to RAG systems and are relevant to modern LLM usage.
Critical weaknesses
-
Limited novelty over CoMem While novelty is a tricky weakness to address and I often try my best not to bring it up, it seems like this work really is mostly applying typical group-wise quantization to CoMem. The fact that this can work is, in my opinion, very expected, as there already exists a ton of KV Cache quantization work, which is highly relevant to activation quantization. There even exist works like XQuant showing that activation quantization is very possible. While their inference pipeline is different from EncBank, it seems fairly trivial to conclude that the activations stored by CoMem can withstand some quantization pressure, especially when the upper layers are open for fine-tuning.
-
Unfair comparison in Table 1 I was very surprised when reading Table 1 and seeing that the proposed EncBank is able to beat Qwen3-8B by such a huge margin. But then I realized these baselines are seeing the full source as context, where the proposed EncBank only sees retrieved sources. This comparison is fundamentally confounded because the BM25-based retriever is independent of EncBank and could also be applied to Dense. This setting renders Table 1 largely uninformative beyond showing that CoMem can be quantized to different budgets.
-
Thin and dated comparison with chunk-reuse methods. One other aspect of this work is to compare with CacheBlend-like work on chunk reuse. This is mostly done in Table 4, using just H16 (which is essentially the CoMem-style setting if my understanding is correct) and only evaluated on two datasets. CacheBlend is an early work in this category, but none of the stronger follow-up methods from the past few years are compared. The same criticism can also be extended to KIVI which is also dated, though I view this as a more minor issue since KV Cache quantization is probably not the most relevant line of work for EncBank.
-
Limited benefit of H4/H8 beyond storage Table 5 seems to suggest that the only major benefit of H4/H8 over H16 is storage. The request-time performance barely improves. More concerningly, EncBank is actually slower than Dense during decoding, even though algorithmically it should not be. This suggests substantial implementation overhead and makes the systems results hard to interpret.
It is possible that I have a major misunderstanding of this work’s positioning, and I am open to changing my mind if that is the case. Based on my current reading, however, the main contribution appears to be applying standard group-wise quantization to CoMem’s stored activations. I do not think combining existing ideas is inherently problematic, but such a combination should usually be technically nontrivial or meaningfully underexplored. Given the extensive literature on KV Cache and activation quantization, I currently do not see that here.
On the empirical evidence side, I feel like the most relevant setting is some kind of RAG evaluation, similar to CacheBlend, where retrieved passages are fixed and different chunk-reuse methods are compared directly. I would simply like to see broader coverage in terms of standard RAG benchmarks, baseline methods, and model backbones.
Questions for authors
- What is the difference between EncBank-H16 and CoMem?
- Why is KIVI slower than Dense in Table 5? I thought the common understanding is that LLM decoding is often memory-bound, in which case quantization should help.
- Since EncBank fine-tune the upper layers of the LLM, would it effect LLM's performance on typical knowledge and reasoning tasks?
- EncBank constructs chunks by their source order, but many sources are naturally orderless. How will EncBank perform under such tasks? Also is there any ablation on such ordering?