- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SOFT-BIT KV, which assigns 1-, 2-, or 8-bit precision to cached tokens under a memory budget. A learned MLP estimates token importance from question-window attention features, and a constrained allocation rule determines precision. The default mode fixes the allocation once; the reload variant stores multiple precision versions and selects among them for each new question.
Critical strengths
- Coherent design that preserves all tokens while allocating more precision to important information.
- The learned importance detector is a meaningful distinction from heuristic mixed-precision methods.
- Promising results against token-dropping and quantization baselines.
- The reload variant provides a practical mechanism for adapting precision to changing questions.
Critical weaknesses
- Unclear generalization. The detector is trained on synthetic factual retrieval, which closely resembles important components of the evaluation benchmarks. Transfer to mathematical reasoning, coding agents, and other task families remains insufficiently evaluated.
- Incomplete compression-rate comparison. SCBench is evaluated primarily at 5.2×, while RULER is evaluated at 11.8×. This prevents separating benchmark sensitivity from compression severity.
- Limited evidence isolating the contribution. Importance-aware mixed precision has substantial prior work. More controlled ablations are needed to establish how much improvement comes from the learned detector versus the allocation rule.
- Limited evaluation of conversational memory. Additional benchmarks such as LoCoMo and LongMemEval would strengthen the practical generalization claim.
Questions for authors
- Can the authors evaluate both SCBench and RULER at both 5.2× and 11.8× compression, ideally with intermediate budgets?
- How well does the retrieval-trained detector transfer to mathematical reasoning and coding-agent benchmarks?
- How sensitive are the results to the detector’s training distribution? Would training on another task family preserve retrieval performance?
- Under a common quantizer and byte budget, how much improvement comes from the MLP versus heuristic scoring, and from constrained allocation versus simpler precision assignment?
- Does the method maintain consistent benefits on LoCoMo and LongMemEval at both compression ratios?
2 comments
Dear Reviewer kqw2,
We thank the reviewer again for the review. We would be grateful to know whether our response addresses the reviewer's concerns, and we are glad to clarify anything further : )
Best regards,
Authors
Dear Reviewer kqw2,
Thanks again for the reviews! We deeply appreciate the time and effort you have invested in evaluating our work :)
We've carefully considered your comments and have prepared a detailed rebuttal addressing each point raised. We are hoping reviewers would be able to review our responses and to engage in a discussion before the end of the discussion period. Please let us know if there are any specific areas you'd like us to clarify further or discuss in more detail. We look forward to your thoughts and are eager to engage in a productive dialogue.
Thank you once again for your commitment to the review process.
Best regards,
Authors