- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
KV retrieval saves attention, but frequent retrieval creates new overhead, so this paper proposed a sentence-level retrieval method, which avoids repeated retreival within a generated sentence.
Critical strengths
- I like the clear problem formulation.
- For the sentence representation, this paper uses special trainable embeddings, which could be more accurate than simple pooling on the keys.
- The hierarchical token selection method is smart: firstly retrieve important sentences, and then pick important tokens inside the retrieved sentences.
Critical weaknesses
- No ablation of fixed-size chunk retrival.
- The overall idea is built on SentenceKV's sentence-level offloading and onloading, but the paper did not mention what is the difference between SentenceKV and KVScout.
- The paper focuses on decoding, not prefilling. But benchmarks like LongBench and RULER are long-input/short-output benchmarks. The reason I give this paper weak rejection is mainly because of this. I hope to see the results of reasoning benchmarks like MATH500 and GPQA-D.
- With reasoning benchmark results and the answer to the question below, I'll raise my score.
Questions for authors
I feel like there is a potential mismatch between the training and inference of the [RET] probe, and hope the author can give some explanations for this.
During training, the paper states that the [RET] probe attends to the full input context together with the generation prefix up to the current sentence boundary. However, during inference, the full context KV cache has been offloaded, and the [RET] probe only attends to the current generation prefix and the GPU-resident window . Therefore, the retrieval query is constructed under different information conditions during training and inference. This raises two concerns. First, during training, the [RET] probe may directly attend to the relevant evidence that it is subsequently trained to retrieve, which could make the contrastive retrieval task easier than the actual predictive retrieval problem encountered at inference. Second, this train-test distribution mismatch may affect the reliability of the reported retriever accuracy if the validation queries are also constructed with access to the full context. It would be helpful for the authors to clarify this discrepancy and evaluate a training setting in which the [RET] probe is conditioned only on a simulated sparse window matching the inference-time . An ablation comparing full-context training against inference-matched sparse-window training would help determine how much this mismatch affects retrieval quality and downstream generation.